In this step-by-step guide, you'll learn how to leverage Immersal's powerful capabilities to seamlessly place and interact with a virtual TV in your real-world environment.
Last updated
Repo: https://github.com/holokit/virtual-tv
In this tutorial, we will guide you through the step-by-step process of achieving the following functionality: placing a virtual TV in your surrounding environment and controlling video playback on this virtual TV using screen taps.
This tutorial is divided into the following sections:
In this chapter, we will introduce the course's topic and the prerequisites, including uploading and downloading Immersal point cloud data, as well as how to download our resource project.
This chapter will cover how to create an AR scene and add the television to the scene.
In this chapter, we will discuss how to use your own point cloud data to replace the point cloud data in the sample scene.
In this chapter, we will learn how to obtain a 3D model of a modern television and import it into the Unity project. We will use the Immersal SDK to capture the surrounding environment's information so that we can accurately place the virtual TV in the desired location.
We will introduce how to add and use a video player component and associate it with a video source so that our TV can play videos.
In this chapter, we will explore how to add interactive elements, such as allowing users to start or stop video playback by clicking on the television.
Once we have completed the creation of our AR scene, we will learn how to debug our application to ensure it runs smoothly on mobile devices.
We will discuss how to build and deploy our AR application to mobile devices so that we can observe and interact with the virtual TV in a real-world environment.
In the conclusion, we will recap the knowledge we have gained throughout the tutorial.
Now that you have an understanding of the chapter structure, let's start with Chapter 1!
Welcome to the "Mastering Immersal: Adding a Virtual Television to Your Room" tutorial. In this first chapter, we will lay the foundation and ensure you have everything you need to embark on this AR journey.
Before we dive into the world of augmented reality televisions, let's make sure you have the prerequisites in place:
PointCloud Data from Immersal
We must capture the surrounding dataset ourselves using Immersal app before start this journey. In this tutorial, we recommend choosing a space that is about the size of a bedroom, and capturing at least 100 images, then uploading. Then downloading the point cloud file through Immersal developer portal.
If you're unsure about how to mapping and upload data to portal, please refer to the documentation: https://immersal.gitbook.io/sdk/tutorials/workflow-and-limitations
Downloading Project
The provided download includes all the necessary resources and a sample scene required for this tutorial.
In this chapter, we will delve into the process of crafting our AR scene in Unity, preparing the stage for placing a modern TV model in your real-world environment.
Overview: To begin, let's create the AR scene in Unity. In this section, we will:
Create a new scene to house our AR experience.
Add HoloKitCameraManager components to the scene to enable Stereo Rendering.
Add HoloKit UI to switch rendering mode.
Steps
Open Unity.
Navigate to folder âImmersalSDKâ > âSamplesâ > âMultimapSampleâ
Right-click to duplicate it into a new scene, then save it with a suitable name, such as "ARSceneâ
Steps
In the Hierarchy Window, find and select the âXR Originâ. Click the gray triangle to the left of XR Origin to open its sub-hierarchy.
Continue by clicking the gray triangle to the left of âCamera Offsetâ to open its sub-hierarchy.
Click on âMain Cameraâ in the sub-hierarchy of âCamera Offsetâ, and go Inspector Window on the right side of the Unity.
Click the "Add Component" button at the bottom, type in "HoloKit Camera Manager" and click on "HoloKit Camera Manager" from the list to apply the component.
Unlike the typical AR scenes, our current scene features two rendering modes. One is the Screen-AR rendering mode, and the other is the Stereo-AR Rendering mode that supports HoloKit hardware. Therefore, we also require a simple button to facilitate the switching between these two modes.
To access it, simply right-click within the Hierarchy Window, hover over "UI" and choose "Default UI Canvas" to create the button.
With these steps, you will have a new Unity scene set up for your AR experience, equipped with the necessary HoloKit AR components.
In the Scene Window, you'll notice there's already a space drawn with a point cloud.
Our next step is to replace this existing point cloud with our custom one. Let's proceed.
Steps
Select "AR Space" and expand its sub-hierarchy. Click on object âAR map 18500-A."
This scene currently uses three point cloud files to reconstruct the environment, which might be more than we need. We only require one point cloud file, and that's what we have. Therefore, let's proceed by deleting "AR Map 18500-B" and "AR Map 18500-C" from the scene.
Click on the "Map File" labeled "18500-A." This action will navigate our Project Window to the folder where this map file is stored. Drag and drop our point cloud file downloaded from immersal portal to this folder. Then use you own file to replace â18500-Aâ.
Thatâs it!
Overview: In this section, we will import a modern TV model into our scene, which will serve as the virtual television that users will interact with.
Download model(with fbx format) from: https://sketchfab.com/3d-models/flatscreen-tv-46-inch-94c7ccaea76f4093b484828419db25cb#download
In Unity, go to "Assets", âRight Clickâ > âCreateâ > âFolderâ to create a folder, rename it to âMeshesâ:
Drag the folder we downloaded into the "Meshes" folder to import all the resources:
Click on the mesh file we imported, we found that inspector window shows thereâs no material on it, because the model and the textures are not well connected:
Let's fix this problem by following steps.
Select the mesh file, go to the Inspector panel, and switch to the Materials section. Then, choose "Extract Materialsâ:
You will find a material created in the folder contains the TV model:
Select the material file, then drag the texture files from the "textures" folder into their respective slots for the Base Map, Metallic Map, Normal Map, Height Map, and Occlusion Map in the material properties:
Now, our TV model has the correct material. Drag the mesh to scene âAR Sceneâ, a TV shows up!
Overview: In this section, we will set up a video source to be displayed on the TV model within the AR scene.
Download the video here: https://www.pexels.com/video/a-river-near-the-rocky-mountains-15135149/ or get a video you like. Ensure the video is in a compatible format, such as .mp4.
Steps
In Unity, go to "Assets", create a folder and rename it to âVideosâ, drag and drop the video file we downloaded to âVideosâ.
Create an Empty Object in your scene, rename it to âVideo playerâ.
Add Video Player component to it.
Drag the video we imported to the field âVideo Clipâ, and check âLoopâ:
Under Videos folder, Right-Click > âCreateâ > âRender Textureâ, rename it to âRT_VideoPlayâ. Select it and set âSizeâ to 1920x1080:
Drag this render texture to the field âTarget Textureâ:
Now, the current frame of the video playback is copied to rendertexture "RT_VideoPlay", we can use "RT_VideoPlay" as a dynamic texture.
Create a new material in the current folder to use for displaying video content, and rename it to "M_Screenâ:
Drag and drop âRT_VideoPlayâ as the Base Map of material âM_Screenâ:
In hierarchy window, âRight Clickâ > âCreate" > â3D Objectâ > âQuadâ to create a Quad, rename it to âScreenâ. Apply material âM_Screenâ to it:
Here, we will search for an appropriate location within the mapped environment to place the TV. Then, we will position the TV at that location in the editor.
By the end of this chapter, you'll have a captivating AR scene with a virtual TV at specific position and displaying your chosen video content.
In my case, I'm writing this tutorial in a serene tea room environmentīŧ
The curved platform here seems like an excellent choice, serving both as a spot for placing the TV and as a relaxation area. Crafting an AR experience in such a setting promises to be quite fascinating.
To achieve this, I need to position the TV within the point cloud space accurately in the Unity editor. Here is how my pointcloud displayed inside Unity:
Within the point cloud, I can easily identify the approximate location for placing the platform:
With these steps, you will have successfully added a modern TV model to your AR scene.
Overview: In this section, we will enable users to interact with the virtual TV model using touch. Users will be able to touch the TV to pause and play the video.
By the end of this chapter, users will have a delightful AR interaction experience with the virtual TV, being able to control video playback through screen tap.
Steps
Add Component "VideoPlayerController" to our "VideoPlayer":
Add Component "ClickDetection" to our "SM_TV". This script makes the object it's attached to "clickable" allowing us to trigger events by tap on it:
Click â+â to create an empty event:
Drag âVideo Playerâ object to empty slot of "Click Detection":
Select function > "VideoPlayerController" > "PlayOrPauseVideo()":
In the end, your Click Detection Component should look like this:
Click on the "Play" button at the top to enter play mode.
Move the camera to ensure that the TV is completely visible in the field of view in the Simulator Window.
Test the pause and play functionality by clicking on the TV screen with the mouse.
Click the play button again at the top to exit play mode.
If everything is working fine, let's move on to deploy our scene to a mobile device!
Building a Unity scene onto your iOS device involves several steps:
Adjust Unity's Build Settings to match your device and requirements.
In Unity, build the project to create a project file for Xcode.
Open this file in Xcode, configure the signing, connect your device via USB, and then click "Run" to build and install the app on your device.
Details of there steps are not covered in this tutorial. If you encounter any issues, you can search for relevant topics like "How to build a Unity project for mobile" to find more instructions.
Here's what we made! You can see that our TV blends well with the environment, which depends on specific environmental settings, layout, lighting conditions, and more. As you become more familiar, you can customize virtual assets that better suit your surroundings.
In this tutorial, we embarked on an exciting journey to create a captivating Augmented Reality (AR) experience using Immersal SDK and Unity. We achieved the remarkable feat of placing a modern television model in the real world and seamlessly integrating a looping video source within it.
As you continue your AR journey, remember that innovation knows no bounds. You now have the foundation to create AR experiences limited only by your imagination. Whether it's enhancing educational content, gamifying real-world exploration, or revolutionizing marketing campaigns, AR offers boundless opportunities to captivate and engage users.
Thank you for joining us on this enlightening journey. We can't wait to see the incredible AR experiences you'll bring to life with Immersal SDK, Unity, and your creative vision. Happy coding and may your AR adventures be truly immersive!
