🎡Tutorial 2: Use Immersal's Visual Positioning System for AR Exhibition

In this step-by-step guide, you'll learn how to leverage Immersal's powerful capabilities to seamlessly place and interact with a virtual TV in your real-world environment.

Repo: https://github.com/holokit/virtual-tv

In this tutorial, we will guide you through the step-by-step process of achieving the following functionality: placing a virtual TV in your surrounding environment and controlling video playback on this virtual TV using screen taps.

Final result

This tutorial is divided into the following sections:

Chapter 1: Preparation

In this chapter, we will introduce the course's topic and the prerequisites, including uploading and downloading Immersal point cloud data, as well as how to download our resource project.

Chapter 2: Creating an AR Scene

This chapter will cover how to create an AR scene and add the television to the scene.

Chapter 3: Replacing Point Cloud Data

In this chapter, we will discuss how to use your own point cloud data to replace the point cloud data in the sample scene.

Chapter 4: Placing the Virtual TV

In this chapter, we will learn how to obtain a 3D model of a modern television and import it into the Unity project. We will use the Immersal SDK to capture the surrounding environment's information so that we can accurately place the virtual TV in the desired location.

Chapter 5: Adding a Video Player

We will introduce how to add and use a video player component and associate it with a video source so that our TV can play videos.

Chapter 6: Interaction and User Experience

In this chapter, we will explore how to add interactive elements, such as allowing users to start or stop video playback by clicking on the television.

Chapter 7: Debugging, Building, and Deploying to Mobile

Once we have completed the creation of our AR scene, we will learn how to debug our application to ensure it runs smoothly on mobile devices.

We will discuss how to build and deploy our AR application to mobile devices so that we can observe and interact with the virtual TV in a real-world environment.

Chapter 8: Conclusion

In the conclusion, we will recap the knowledge we have gained throughout the tutorial.

Now that you have an understanding of the chapter structure, let's start with Chapter 1!

Chapter 1: Getting Started

Welcome to the "Mastering Immersal: Adding a Virtual Television to Your Room" tutorial. In this first chapter, we will lay the foundation and ensure you have everything you need to embark on this AR journey.

Prerequisites

Before we dive into the world of augmented reality televisions, let's make sure you have the prerequisites in place:

1. PointCloud Data from Immersal

   We must capture the surrounding dataset ourselves using Immersal app before start this journey. In this tutorial, we recommend choosing a space that is about the size of a bedroom, and capturing at least 100 images, then uploading. Then downloading the point cloud file through Immersal developer portal.

   

   download data with .bytes format

   If you're unsure about how to mapping and upload data to portal, please refer to the documentation: https://immersal.gitbook.io/sdk/tutorials/workflow-and-limitations

2. Downloading Project

   The provided download includes all the necessary resources and a sample scene required for this tutorial.

Chapter 2: Creating the AR Scene

In this chapter, we will delve into the process of crafting our AR scene in Unity, preparing the stage for placing a modern TV model in your real-world environment.

Overview: To begin, let's create the AR scene in Unity. In this section, we will:

1. Create a new scene to house our AR experience.

2. Add HoloKitCameraManager components to the scene to enable Stereo Rendering.

3. Add HoloKit UI to switch rendering mode.

2.1: Creating a New Scene

Steps

1. Open Unity.

2. Navigate to folder “ImmersalSDK” > “Samples” > “MultimapSample”

3. Right-click to duplicate it into a new scene, then save it with a suitable name, such as "ARScene”

   

   Duplicate a scene, rename to "ARScene"

2.2: Adding HoloKit AR Components

Steps

1. In the Hierarchy Window, find and select the “XR Origin”. Click the gray triangle to the left of XR Origin to open its sub-hierarchy.

2. Continue by clicking the gray triangle to the left of “Camera Offset” to open its sub-hierarchy.

3. Click on “Main Camera” in the sub-hierarchy of “Camera Offset”, and go Inspector Window on the right side of the Unity.

4. Click the "Add Component" button at the bottom, type in "HoloKit Camera Manager" and click on "HoloKit Camera Manager" from the list to apply the component.

   

   Add HoloKitCameraManager Component

2.3 Adding a Button for Rendering Mode Switching

Unlike the typical AR scenes, our current scene features two rendering modes. One is the Screen-AR rendering mode, and the other is the Stereo-AR Rendering mode that supports HoloKit hardware. Therefore, we also require a simple button to facilitate the switching between these two modes.

To access it, simply right-click within the Hierarchy Window, hover over "UI" and choose "Default UI Canvas" to create the button.

Add HoloKit UI Canvas

With these steps, you will have a new Unity scene set up for your AR experience, equipped with the necessary HoloKit AR components.

Section 3: Replacing Pointcloud

In the Scene Window, you'll notice there's already a space drawn with a point cloud.

Pointcloud displayed inside scene window

Our next step is to replace this existing point cloud with our custom one. Let's proceed.

Steps

1. Select "AR Space" and expand its sub-hierarchy. Click on object “AR map 18500-A."

2. This scene currently uses three point cloud files to reconstruct the environment, which might be more than we need. We only require one point cloud file, and that's what we have. Therefore, let's proceed by deleting "AR Map 18500-B" and "AR Map 18500-C" from the scene.

   

   Delete unnecessary "AR Map" objects

3. Click on the "Map File" labeled "18500-A." This action will navigate our Project Window to the folder where this map file is stored. Drag and drop our point cloud file downloaded from immersal portal to this folder. Then use you own file to replace “18500-A”.

   

   Replace point cloud file with your own

   That’s it!

Section 4: Placing the TV Model

Overview: In this section, we will import a modern TV model into our scene, which will serve as the virtual television that users will interact with.

4.1: Obtaining a 3D TV Model

Download model(with fbx format) from: https://sketchfab.com/3d-models/flatscreen-tv-46-inch-94c7ccaea76f4093b484828419db25cb#download

Download model from sketchfab.com

4.2: Importing the TV Model

1. In Unity, go to "Assets", “Right Click” > “Create” > “Folder” to create a folder, rename it to “Meshes”:

   

   Create "Meshes" folder
2. Drag the folder we downloaded into the "Meshes" folder to import all the resources:

   

   Make sure you have imported both folders, "source" and "texture," to ensure you have all the necessary files

Click on the mesh file we imported, we found that inspector window shows there’s no material on it, because the model and the textures are not well connected:

No materials on TV

Let's fix this problem by following steps.

1. Select the mesh file, go to the Inspector panel, and switch to the Materials section. Then, choose "Extract Materials”:

   

   Extract Materials

   You will find a material created in the folder contains the TV model:

   

   Material "M_FlatScreen"
2. Select the material file, then drag the texture files from the "textures" folder into their respective slots for the Base Map, Metallic Map, Normal Map, Height Map, and Occlusion Map in the material properties:

   

Now, our TV model has the correct material. Drag the mesh to scene “AR Scene”, a TV shows up!

TV with material

Section 5: Adding a Video Source

Overview: In this section, we will set up a video source to be displayed on the TV model within the AR scene.

5.1: Preparing Video Content

Download the video here: https://www.pexels.com/video/a-river-near-the-rocky-mountains-15135149/ or get a video you like. Ensure the video is in a compatible format, such as .mp4.

Click "Free Download" to download video

5.2: Setting Up Video Playback

Steps

1. In Unity, go to "Assets", create a folder and rename it to “Videos”, drag and drop the video file we downloaded to “Videos”.

   

   Create folder and add video file

2. Create an Empty Object in your scene, rename it to “Video player”.

3. Add Video Player component to it.

   

   Add Video Player component

4. Drag the video we imported to the field “Video Clip”, and check “Loop”:

   

   Add Video Clip and check loop option

5. Under Videos folder, Right-Click > “Create” > “Render Texture”, rename it to “RT_VideoPlay”. Select it and set “Size” to 1920x1080:

   

   Set size to 1920*1080

6. Drag this render texture to the field “Target Texture”:

   

   Add Target Texture

Now, the current frame of the video playback is copied to rendertexture "RT_VideoPlay", we can use "RT_VideoPlay" as a dynamic texture.

5.3 Set the TV screen texture to display the video

Steps

1. Create a new material in the current folder to use for displaying video content, and rename it to "M_Screen”:

   

   Create a material and rename it to "M_Screen"

2. Drag and drop “RT_VideoPlay” as the Base Map of material “M_Screen”:

   

   Add Base Map

3. In hierarchy window, “Right Click” > ”Create" > “3D Object” > “Quad” to create a Quad, rename it to “Screen”. Apply material “M_Screen” to it:

   

   Create a Quad and add material
4. Make “Screen” a child of SM_TV, move the position and make sure it precisely covers the TV screen's position and scale to a same size of screen:

   

   Move and scale "Screen" object

5.4 Placing TV to our RealWorld

Here, we will search for an appropriate location within the mapped environment to place the TV. Then, we will position the TV at that location in the editor.

By the end of this chapter, you'll have a captivating AR scene with a virtual TV at specific position and displaying your chosen video content.

In my case, I'm writing this tutorial in a serene tea room environment：

A serene tea room

The curved platform here seems like an excellent choice, serving both as a spot for placing the TV and as a relaxation area. Crafting an AR experience in such a setting promises to be quite fascinating.

To achieve this, I need to position the TV within the point cloud space accurately in the Unity editor. Here is how my pointcloud displayed inside Unity:

My point cloud shows in Unity

Within the point cloud, I can easily identify the approximate location for placing the platform:

Placing the TV on the curved platform

With these steps, you will have successfully added a modern TV model to your AR scene.

Chapter 6: Implementing AR Interaction

Overview: In this section, we will enable users to interact with the virtual TV model using touch. Users will be able to touch the TV to pause and play the video.

By the end of this chapter, users will have a delightful AR interaction experience with the virtual TV, being able to control video playback through screen tap.

Steps

1. Add Component "VideoPlayerController" to our "VideoPlayer":

   

   Click Add Component and search for "VideoPlayerController"
2. Add Component "ClickDetection" to our "SM_TV". This script makes the object it's attached to "clickable" allowing us to trigger events by tap on it:

   

   Click Add Component and search for "ClickDetection"
3. Click “+” to create an empty event:

   
4. Drag “Video Player” object to empty slot of "Click Detection":

   
5. Select function > "VideoPlayerController" > "PlayOrPauseVideo()":

   

   In the end, your Click Detection Component should look like this:

   

Chapter 7: Testing, Building and Deploying for iOS

7.1 Testing In Editor

1. Click on the "Play" button at the top to enter play mode.

   
2. Move the camera to ensure that the TV is completely visible in the field of view in the Simulator Window.

   
3. Test the pause and play functionality by clicking on the TV screen with the mouse.

4. Click the play button again at the top to exit play mode.

If everything is working fine, let's move on to deploy our scene to a mobile device!

7.2 Building and Deploying

Building a Unity scene onto your iOS device involves several steps:

1. Adjust Unity's Build Settings to match your device and requirements.

2. In Unity, build the project to create a project file for Xcode.

3. Open this file in Xcode, configure the signing, connect your device via USB, and then click "Run" to build and install the app on your device.

Details of there steps are not covered in this tutorial. If you encounter any issues, you can search for relevant topics like "How to build a Unity project for mobile" to find more instructions.

Here's what we made! You can see that our TV blends well with the environment, which depends on specific environmental settings, layout, lighting conditions, and more. As you become more familiar, you can customize virtual assets that better suit your surroundings.

Chapter 8: Conclusion

In this tutorial, we embarked on an exciting journey to create a captivating Augmented Reality (AR) experience using Immersal SDK and Unity. We achieved the remarkable feat of placing a modern television model in the real world and seamlessly integrating a looping video source within it.

As you continue your AR journey, remember that innovation knows no bounds. You now have the foundation to create AR experiences limited only by your imagination. Whether it's enhancing educational content, gamifying real-world exploration, or revolutionizing marketing campaigns, AR offers boundless opportunities to captivate and engage users.

Thank you for joining us on this enlightening journey. We can't wait to see the incredible AR experiences you'll bring to life with Immersal SDK, Unity, and your creative vision. Happy coding and may your AR adventures be truly immersive!