Comment on page
Tutorial 2: Use Immersal's Visual Positioning System for AR Exhibition
In this step-by-step guide, you'll learn how to leverage Immersal's powerful capabilities to seamlessly place and interact with a virtual TV in your real-world environment.
In this tutorial, we will guide you through the step-by-step process of achieving the following functionality: placing a virtual TV in your surrounding environment and controlling video playback on this virtual TV using screen taps.
This tutorial is divided into the following sections:
In this chapter, we will introduce the course's topic and the prerequisites, including uploading and downloading Immersal point cloud data, as well as how to download our resource project.
This chapter will cover how to create an AR scene and add the television to the scene.
In this chapter, we will discuss how to use your own point cloud data to replace the point cloud data in the sample scene.
In this chapter, we will learn how to obtain a 3D model of a modern television and import it into the Unity project. We will use the Immersal SDK to capture the surrounding environment's information so that we can accurately place the virtual TV in the desired location.
We will introduce how to add and use a video player component and associate it with a video source so that our TV can play videos.
In this chapter, we will explore how to add interactive elements, such as allowing users to start or stop video playback by clicking on the television.
Once we have completed the creation of our AR scene, we will learn how to debug our application to ensure it runs smoothly on mobile devices.
We will discuss how to build and deploy our AR application to mobile devices so that we can observe and interact with the virtual TV in a real-world environment.
In the conclusion, we will recap the knowledge we have gained throughout the tutorial.
Now that you have an understanding of the chapter structure, let's start with Chapter 1!
Welcome to the "Mastering Immersal: Adding a Virtual Television to Your Room" tutorial. In this first chapter, we will lay the foundation and ensure you have everything you need to embark on this AR journey.
Before we dive into the world of augmented reality televisions, let's make sure you have the prerequisites in place:
- 1.PointCloud Data from ImmersalWe must capture the surrounding dataset ourselves using Immersal app before start this journey. In this tutorial, we recommend choosing a space that is about the size of a bedroom, and capturing at least 100 images, then uploading. Then downloading the point cloud file through Immersal developer portal.download data with .bytes formatIf you're unsure about how to mapping and upload data to portal, please refer to the documentation: https://immersal.gitbook.io/sdk/tutorials/workflow-and-limitations
- 2.Downloading ProjectDownload tutorial project directly from: https://github.com/holoi/holokit-tutorial-002The provided download includes all the necessary resources and a sample scene required for this tutorial.
In this chapter, we will delve into the process of crafting our AR scene in Unity, preparing the stage for placing a modern TV model in your real-world environment.
Overview: To begin, let's create the AR scene in Unity. In this section, we will:
- 1.Create a new scene to house our AR experience.
- 2.Add HoloKitCameraManager components to the scene to enable Stereo Rendering.
- 3.Add HoloKit UI to switch rendering mode.
- 1.Open Unity.
- 2.Navigate to folder “ImmersalSDK” > “Samples” > “MultimapSample”
- 3.Right-click to duplicate it into a new scene, then save it with a suitable name, such as "ARScene”Duplicate a scene, rename to "ARScene"
- 1.In the Hierarchy Window, find and select the “XR Origin”. Click the gray triangle to the left of XR Origin to open its sub-hierarchy.
- 2.Continue by clicking the gray triangle to the left of “Camera Offset” to open its sub-hierarchy.
- 3.Click on “Main Camera” in the sub-hierarchy of “Camera Offset”, and go Inspector Window on the right side of the Unity.
- 4.Click the "Add Component" button at the bottom, type in "HoloKit Camera Manager" and click on "HoloKit Camera Manager" from the list to apply the component.Add HoloKitCameraManager Component
Unlike the typical AR scenes, our current scene features two rendering modes. One is the Screen-AR rendering mode, and the other is the Stereo-AR Rendering mode that supports HoloKit hardware. Therefore, we also require a simple button to facilitate the switching between these two modes.
To access it, simply right-click within the Hierarchy Window, hover over "UI" and choose "Default UI Canvas" to create the button.
Add HoloKit UI Canvas
With these steps, you will have a new Unity scene set up for your AR experience, equipped with the necessary HoloKit AR components.
In the Scene Window, you'll notice there's already a space drawn with a point cloud.
Pointcloud displayed inside scene window
Our next step is to replace this existing point cloud with our custom one. Let's proceed.
- 1.Select "AR Space" and expand its sub-hierarchy. Click on object “AR map 18500-A."
- 2.This scene currently uses three point cloud files to reconstruct the environment, which might be more than we need. We only require one point cloud file, and that's what we have. Therefore, let's proceed by deleting "AR Map 18500-B" and "AR Map 18500-C" from the scene.Delete unnecessary "AR Map" objects
- 3.Click on the "Map File" labeled "18500-A." This action will navigate our Project Window to the folder where this map file is stored. Drag and drop our point cloud file downloaded from immersal portal to this folder. Then use you own file to replace “18500-A”.Replace point cloud file with your ownThat’s it!
Overview: In this section, we will import a modern TV model into our scene, which will serve as the virtual television that users will interact with.
Download model from sketchfab.com
- 1.In Unity, go to "Assets", “Right Click” > “Create” > “Folder” to create a folder, rename it to “Meshes”:Create "Meshes" folder
- 2.Drag the folder we downloaded into the "Meshes" folder to import all the resources:Make sure you have imported both folders, "source" and "texture," to ensure you have all the necessary files
Click on the mesh file we imported, we found that inspector window shows there’s no material on it, because the model and the textures are not well connected:
No materials on TV
Let's fix this problem by following steps.
- 1.Select the mesh file, go to the Inspector panel, and switch to the Materials section. Then, choose "Extract Materials”:Extract MaterialsYou will find a material created in the folder contains the TV model:Material "M_FlatScreen"
- 2.Select the material file, then drag the texture files from the "textures" folder into their respective slots for the Base Map, Metallic Map, Normal Map, Height Map, and Occlusion Map in the material properties:
Now, our TV model has the correct material. Drag the mesh to scene “AR Scene”, a TV shows up!
TV with material
Overview: In this section, we will set up a video source to be displayed on the TV model within the AR scene.
Click "Free Download" to download video
- 1.In Unity, go to "Assets", create a folder and rename it to “Videos”, drag and drop the video file we downloaded to “Videos”.Create folder and add video file
- 2.Create an Empty Object in your scene, rename it to “Video player”.
- 3.Add Video Player component to it.Add Video Player component
- 4.Drag the video we imported to the field “Video Clip”, and check “Loop”:Add Video Clip and check loop option
- 5.Under Videos folder, Right-Click > “Create” > “Render Texture”, rename it to “RT_VideoPlay”. Select it and set “Size” to 1920x1080:Set size to 1920*1080
- 6.Drag this render texture to the field “Target Texture”:Add Target Texture
Now, the current frame of the video playback is copied to rendertexture "RT_VideoPlay", we can use "RT_VideoPlay" as a dynamic texture.
- 1.Create a new material in the current folder to use for displaying video content, and rename it to "M_Screen”:Create a material and rename it to "M_Screen"
- 2.Drag and drop “RT_VideoPlay” as the Base Map of material “M_Screen”:Add Base Map
- 3.In hierarchy window, “Right Click” > ”Create" > “3D Object” > “Quad” to create a Quad, rename it to “Screen”. Apply material “M_Screen” to it:Create a Quad and add material
Here, we will search for an appropriate location within the mapped environment to place the TV. Then, we will position the TV at that location in the editor.
By the end of this chapter, you'll have a captivating AR scene with a virtual TV at specific position and displaying your chosen video content.
In my case, I'm writing this tutorial in a serene tea room environment：
A serene tea room
The curved platform here seems like an excellent choice, serving both as a spot for placing the TV and as a relaxation area. Crafting an AR experience in such a setting promises to be quite fascinating.
To achieve this, I need to position the TV within the point cloud space accurately in the Unity editor. Here is how my pointcloud displayed inside Unity:
My point cloud shows in Unity
Within the point cloud, I can easily identify the approximate location for placing the platform:
Placing the TV on the curved platform
With these steps, you will have successfully added a modern TV model to your AR scene.
Overview: In this section, we will enable users to interact with the virtual TV model using touch. Users will be able to touch the TV to pause and play the video.
By the end of this chapter, users will have a delightful AR interaction experience with the virtual TV, being able to control video playback through screen tap.
- 1.Add Component "VideoPlayerController" to our "VideoPlayer":Click Add Component and search for "VideoPlayerController"
- 2.Add Component "ClickDetection" to our "SM_TV". This script makes the object it's attached to "clickable" allowing us to trigger events by tap on it:Click Add Component and search for "ClickDetection"
- 3.Click “+” to create an empty event:
- 4.Drag “Video Player” object to empty slot of "Click Detection":
- 5.Select function > "VideoPlayerController" > "PlayOrPauseVideo()":In the end, your Click Detection Component should look like this:
- 1.Click on the "Play" button at the top to enter play mode.
- 2.Move the camera to ensure that the TV is completely visible in the field of view in the Simulator Window.
- 3.Test the pause and play functionality by clicking on the TV screen with the mouse.
- 4.Click the play button again at the top to exit play mode.
If everything is working fine, let's move on to deploy our scene to a mobile device!
Building a Unity scene onto your iOS device involves several steps:
- 1.Adjust Unity's Build Settings to match your device and requirements.
- 2.In Unity, build the project to create a project file for Xcode.
- 3.Open this file in Xcode, configure the signing, connect your device via USB, and then click "Run" to build and install the app on your device.
Details of there steps are not covered in this tutorial. If you encounter any issues, you can search for relevant topics like "How to build a Unity project for mobile" to find more instructions.
Here's what we made! You can see that our TV blends well with the environment, which depends on specific environmental settings, layout, lighting conditions, and more. As you become more familiar, you can customize virtual assets that better suit your surroundings.
In this tutorial, we embarked on an exciting journey to create a captivating Augmented Reality (AR) experience using Immersal SDK and Unity. We achieved the remarkable feat of placing a modern television model in the real world and seamlessly integrating a looping video source within it.
As you continue your AR journey, remember that innovation knows no bounds. You now have the foundation to create AR experiences limited only by your imagination. Whether it's enhancing educational content, gamifying real-world exploration, or revolutionizing marketing campaigns, AR offers boundless opportunities to captivate and engage users.
Thank you for joining us on this enlightening journey. We can't wait to see the incredible AR experiences you'll bring to life with Immersal SDK, Unity, and your creative vision. Happy coding and may your AR adventures be truly immersive!