---
description: >-
  In this tutorial, we will explore the foundational concepts of creating AR
  applications while integrating HoloKit's holographic rendering and
  hand-tracking features.
---

# Tutorial 1: Use Hand Tracking for Interacting with Holograms

See [https://github.com/holokit/touching-hologram](https://github.com/holokit/touching-hologram)

## Overview

This tutorial comprises the following sections:

**Section 1: Preparation**

Embark on your AR journey by learning to download essential project files and open them in Unity.

**Section 2: Scene Setup and Basic AR Components**

Dive into Unity's Scene concept, crafting your very own Scene. Discover the fundamental components necessary for building AR experiences, including XR Origin and AR Session.

**Section 3: Exploring Prefabs**

Delve into the Unity's Prefab concept. Uncover valuable insights into the provided Prefabs featured in this lesson.

**Section 4: Placing Virtual Objects in Real-World Environments**

Master the art of utilizing the AR Raycast Manager to detect and actively position virtual objects within real-world environments.

**Section 5: Unveiling Hand-Tracking Magic**

Experience the magic of HoloKit's Hand Tracking functionality. Elevate your application by introducing natural interaction.

**Section 6: Creating Your SceneManager**

Learn about Unity's SceneManager and create a customized version to efficiently manage your scenes.

**Section 7: Immersive Soundscapes**

Enhance the overall immersion of your scene by integrating background music and sound effects.

**Section 8: Test Run in Play Mode**

Before taking your creation to mobile devices, test it within the Unity editor to ensure a seamless experience.

**Section 9: Building for Mobile Devices**

Collaborate to transfer your scene onto mobile devices and witness the results of your AR journey.

This tutorial will equip you with the skills needed to create holographic AR applications. Let's dive into this thrilling learning journey!

## Section 1：Preparation

#### 1.1 Set Up Unity

The course's accompanying project uses Unity 2022.3.7f1. We recommend you download this version.

Working with other versions of Unity may leads to compatibility issues.

#### 1.2 Download the Course Project

In this tutorial, we will provide a Git project, which is a Unity project containing all the necessary assets and an implemented scene.

We will use this project as our starting point, following the tutorial step by step, to gradually build our own application.

To begin, head over to the project's GitHub page：

[https://github.com/holokit/touching-hologram](https://github.com/holokit/touching-hologram)

There are two ways to download this project, choose either one:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FH0iMTiLNjl3ltmrZEl2d%2Fimage.png?alt=media&#x26;token=71586d8a-4013-42f2-ad37-e5c8ffbd420b" alt=""><figcaption><p>Click on the download to start the download process</p></figcaption></figure>

1. Click the green button to download the zip file. Once the download is complete, locate the compressed file and extract it.
2. If you are familiar with Git, clone this project on your own.

#### 1.3 Open the Project

Once the download is complete, open Unity Hub. Click “Open”:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FdRBru3duXTXIMhm4mnAV%2Fimage.png?alt=media&#x26;token=32c1daf2-2d6b-4d12-baed-91d3ecbe6dbf" alt=""><figcaption><p>Click “Open” to open project folder</p></figcaption></figure>

Select the appropriate folder, and click "Open":

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FGcDX43g8XJNX8SmzZVmN%2Fimage.png?alt=media&#x26;token=3279e832-943a-4695-9476-bd0be0f1a501" alt=""><figcaption><p> Select your project folder</p></figcaption></figure>

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FEiKlCW6nvNTP4JBlGcK4%2Fimage.png?alt=media&#x26;token=0acf300a-c5d0-4fa6-88f8-9939ab0c66fa" alt=""><figcaption><p>Remember to choose Unity 2022.3.7f1 and IOS platform.</p></figcaption></figure>

Wait for Unity to open.

#### 1.4 Configuring Unity Layout

Before delving into the creation of our AR app, we suggest utilizing the "holokit-layout" that we've provided. This layout encompasses all essential windows to ensure a comprehensive workflow.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FldqAVwddM0elfRKgFQIL%2Fimage.png?alt=media&#x26;token=5c3fbb83-d531-44e2-afe2-3efd02baf109" alt=""><figcaption><p>Select the "holokit-layout" from the dropdown menu located in the upper right corner.</p></figcaption></figure>

All set, let's dive into MR development with this layout at our fingertips.

## Section 2: Setup scene and basic AR component

### 2.1 Scene Creation

In Unity, a "Scene" is a fundamental building block that encapsulates a distinct environment within your project. It serves as a self-contained space where you design and construct specific aspects of your game or application. Think of it as a contained setting, much like a chapter in a story, where you create and organize various elements to achieve your desired experience.

Here's how we dive into the action:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2F1AeQfx5WuGQ10t2ycqjT%2F1.avif?alt=media&#x26;token=b6aa7acb-5ced-4fbb-8113-fcb3abdbe332" alt=""><figcaption><p>Create a new scene</p></figcaption></figure>

1. Locate the Project Window. You can find it on the bottom left of the Unity editor.
2. Click on the "My Assets" folder in the Project Window to select it.
3. Right-click and hover over the ”Create” option in the context menu, then select ”Scene” from the submenu. This will create a new empty scene in the project.
4. Right-click, select “Rename”, to rename it to “Buddha”
5. Double-click it to open.

After opening it, you'll notice a default skybox in the Simulator Window as the following image shows:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FNhK85UIx1TgGyAE8WIBz%2Fimage.png?alt=media&#x26;token=97baa498-a3d2-4333-8c67-3915cf47bcc5" alt=""><figcaption><p>Unity Editor: Open Our Scene</p></figcaption></figure>

Next, let's remove all the objects in the Hierarchy Window, including the Camera and Directional Light, for the following reasons:

* We won't be needing the default camera as we'll create new cameras for the Mixed Reality (MR) experience.
* The Directional Light is unnecessary because we won't be illuminating any objects in this tutorial.

### 2.2 Integrating AR Session and XR Origin

When it comes to Augmented Reality (AR) development, Unity's `AR Session` and `XR Origin` play crucial roles:

`AR Session` is one of the primary components in Unity used to manage the AR experience. In essence, it forms the foundation for developing AR applications in Unity, providing interfaces with **AR hardware devices** and tracking systems, as well as managing interactions between virtual content and the real world.

{% hint style="info" %}
AR hardware devices Includes sensors such as gyroscope, accelerometer, magnetometer, and camera. These sensors can be used to obtain information about the device's orientation, direction, acceleration, and position.
{% endhint %}

`XR Origin` is a virtual coordinate system used to define the origin point and axis directions in VR or AR scenes. Understanding and correctly setting up the XR Origin are essential to ensure proper positioning and movement of XR applications in VR or AR. Unless there are specific requirements, always remember to keep the object XR Origin's `Transform` component in its default state (Position(0,0,0), Rotation(0,0,0), Scale(1,1,1)).

{% hint style="info" %}
By default, the XR Origin's position is typically set at the origin of the real world (the position when the mobile application is launched), aligning the virtual and real worlds perfectly. If the XR Origin's position is not the default value, the position of virtual objects will deviate from the actual position in the real world.
{% endhint %}

Let's add these two components to our Scene.

*   Let's start by adding `AR Session`. In the Hierarchy Window, right-click on an empty area → Create → XR → AR Session.

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FZvp9CV5nUAS2Rfy9QEId%2Fimage.png?alt=media&#x26;token=82b79c26-9581-445f-bb30-e1bb81489a6f" alt=""><figcaption><p>Add AR Session</p></figcaption></figure>
*   Next, let's add `XR Origin`. In the Hierarchy Window, double-click on an empty area → Create → XR → XR Origin:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FveKxLNSG4pRUkEVN5KDN%2Fimage.png?alt=media&#x26;token=bbd672c5-a922-4617-9e73-bbf75c788693" alt=""><figcaption><p>Add XR Origin</p></figcaption></figure>

Your Hierarchy Window should look like this:

<div align="center" data-full-width="true"><figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FECWuLkkERcQBz2ClxQRV%2Fimage.png?alt=media&#x26;token=e53b892b-9df5-4c5d-a93c-0df64f3305c9" alt="" width="188"><figcaption><p>Hierarchy Window</p></figcaption></figure></div>

### 2.3 HoloKit Camera and Stereo Rendering

Unity's AR Foundation supports the development of Screen AR applications. However, when we intend to build Stereo AR applications work with HoloKit, we need to utilize the `HoloKitCameraManager`.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FjYwx3ABoyd5ZUM7unTpj%2F2.avif?alt=media&#x26;token=c0ccb1c1-c79b-4de7-b15a-108aba24fc6b" alt=""><figcaption><p>Add HoloKitCameraManager</p></figcaption></figure>

1. In the Hierarchy Window, find and select the “XR Origin”. Click the gray triangle to the left of XR Origin to open its sub-hierarchy.
2. Continue by clicking the gray triangle to the left of “Camera Offset” to open its sub-hierarchy.
3. Click on “Main Camera” in the sub-hierarchy of “Camera Offset”, and go Inspector Window on the right side of the Unity.
4. Click the "Add Component" button at the bottom, type in "HoloKit Camera Manager" and click on "HoloKit Camera Manager" from the list to apply the component.

We could see the `HoloKitCameraManager` component in the Inspector Window:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FDQS3qDSVwrYkW7VDSPsW%2Fimage.png?alt=media&#x26;token=ea2feaaf-0f3c-40c7-896c-b20cf3e706ae" alt=""><figcaption><p>Add HoloKitCameraManager</p></figcaption></figure>

Furthermore, under the “HoloKit Camera”, the `HoloKitCameraManager` help us added the following sub-hierarchy automatically as the following image shows.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FxVz5QnshLw7jKenR8oII%2Fimage.png?alt=media&#x26;token=f688730e-a6cb-4106-99be-50051245c36c" alt=""><figcaption><p>Check Hierarchy</p></figcaption></figure>

Among them, the “Center Eye Pose” represents the point between the two eyes, which can serve as a reference for the movement of our head. “Black Camera”, “Left Eye Camera”, and “Right Eye Camera” are used for background rendering, left eye content rendering, and right eye content rendering, respectively, in Stereo Rendering mode.

#### 2.3.1 Quick Creation of XR Origin with HoloKit Camera

We've also included a menu option that lets you create an “XR Origin” along with a `HoloKitCameraManager` simultaneously, streamlining the process.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FOJj6X2TGjzLoSGsj2mYn%2F3.avif?alt=media&#x26;token=c734ef81-f429-45c6-a010-24e839ccd85b" alt=""><figcaption></figcaption></figure>

* In the empty space within the Hierarchy Window, right-click and navigate to XR → HoloKit → XR Origin.

### 2.4 Adding a Button for Rendering Mode Switching

Unlike the typical AR scenes, our current scene features two rendering modes. One is the Screen-AR rendering mode, and the other is the Stereo-AR Rendering mode that supports HoloKit hardware. Therefore, we also require a simple button to facilitate the switching between these two modes.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FweAGJVtr8bpNb1aDPYiv%2F4.avif?alt=media&#x26;token=48585b0e-73c4-44e1-9b44-39064ad1ec8c" alt=""><figcaption><p>Add button to switch rendering modes</p></figcaption></figure>

* To access it, simply right-click within the Hierarchy Window, hover over "UI" and choose "Default UI Canvas" to create the button.

## Section 3: Explore Prefab

Let's take a look at the virtual object we're about to place in the space — the Prefab Buddha.

The Prefab Buddha is presented in the form of a prefab. Why do we construct a prefab? It's because I want each of us to be able to drag and drop it instantly, making it incredibly convenient.

{% hint style="info" %}
In Unity, a Prefab is an essential resource type that allows for easy reuse. It serves as a template for storing and managing game objects and their components. Prefabs enable you to create and instantiate game objects with identical attributes and components in a scene, making it simple to reuse and manage these objects.
{% endhint %}

First, locate our prefab and drag it into our scene.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2F8kEKrPmUfO1RXaF2hQoK%2F5.avif?alt=media&#x26;token=38f3ee2d-9556-4943-abb4-35fe173eced3" alt=""><figcaption></figcaption></figure>

* Go to the Project Window, open: “Course Assets” → “Course1” → “Prefabs” → “Buddha”
* Click and drag the “Buddha“ into empty area in the Hierarchy Window to place it in the scene for viewing.

The Prefab Buddha consists of three main parts: Visual Part,

*   It has a Visual Effect Component that creates a particle-style representation of the Buddha figure:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FkVRA7wpirzUXvSrVCXp5%2Fimage.png?alt=media&#x26;token=80df5d18-227f-467a-9883-aacf30c934a3" alt=""><figcaption><p>Visual Effect Component</p></figcaption></figure>
*   It has a `Buddha Controller` component that actively searches for the position of hand after this prefab is created and binds the position to the VFX, in this way the Visual Part can interact with our hand:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FwejrEAHHrHzq6Y2ozFbf%2Fimage.png?alt=media&#x26;token=d1b77ac8-d38b-4293-a47b-9b4f9dc76b32" alt=""><figcaption><p>Buddha Controller component</p></figcaption></figure>
*   It has an `AudioSource` component to play sound. When the Buddha is created, play a trigger sound as feedback and enhance the immersive experience:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FmKqLYILLoFoIDFzJJcfn%2Fimage.png?alt=media&#x26;token=5a292396-2b5c-4c6f-bcd0-239a3cd232cd" alt=""><figcaption><p>Audio Source Component</p></figcaption></figure>

The Prefab Buddha is intended to be created at runtime, let's remove the Buddha from the scene for now.

*   Right-click on “Buddha” → “Delete”, to remove it from the scene:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FXGueoNH1uMIF9hwhbHxR%2Fimage.png?alt=media&#x26;token=897fb392-5dbd-4328-b22a-0e19656bc0b2" alt=""><figcaption><p>Delete Buddha From Scene</p></figcaption></figure>

## Section 4: Placing Objects Within AR Environments

### 4.1 ARRaycastManager

Since we aim to position virtual objects within the real world, it's crucial that we have the ability to specify their placement location. Ideally, this location should be contextually relevant to the real environment, rather than consistently floating in mid-air or getting stuck within surfaces like the ground or walls.

To achieve this, `ARRaycastManager` is our go-to choice. it allows you to cast rays in environment to detect planes, objects, feature points, or other virtual content within the AR session.

Let’s add `ARRaycastManager` to our scene.

*   Go to the Hierarchy Window, click on “XR Origin” to select it, click “Add Component”, type in “AR Raycast Manager”, click on `ARRaycastManager` to apply it.

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FuEssdIIN7mPM4AYomEC4%2Fimage.png?alt=media&#x26;token=fc0b5a6c-0e45-4740-90d2-2e8c4225b095" alt=""><figcaption><p>Add AR Raycast Manager</p></figcaption></figure>

Moreover, we require a component to unlock the potential of the `ARRaycastManager`.

The `ARPlacementWithTouch` component leverages the functionalities provided by the `ARRaycastManager`, projecting an invisible ray from the screen's center along the camera's direction. When the user taps the screen and this ray intersects with a plane (like the ground, wall, or ceiling), it instantly creates a predefined virtual object at the intersection point.

Add `ARPlacementWithTouch` to our scene.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2Fl3bdsuX18DSe9W5yzyMB%2F6.avif?alt=media&#x26;token=131d6093-4aff-4724-92c1-0d19fe0730c4" alt=""><figcaption></figcaption></figure>

* First, in the Hierarchy Window, right-click → “Create Empty” to create a GameObject.
* Right-click on it → “Rename”, to rename it to "ARPlacementManager“.
* In the Inspector Window, click “Add Component”, type "ARPlacementWithTouch", click on `ARPlacementWithTouch` to apply it.

In order to inform `ARPlacementWithTouch` about the specific object we want to create, we need to find the prefab in the Project Window and drag it into the "SpawnedPrefab" field of `ARPlacementWithTouch`.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2Fbyte8DNj4PquyMr9YW13%2F7.avif?alt=media&#x26;token=15caf261-166d-42d6-b58d-69f9fbbdfc50" alt=""><figcaption></figcaption></figure>

* Go to the Project Window and find prefab "Buddha".
* Click and drag it into the "SpawnedPrefab" field of `ARPlacementWithTouch`.

Now, whenever you tap the screen on your phone, the virtual object will be created at the specified location. Enjoy the magic of placing virtual objects in the real world!

### 4.2 Placement Indicator

As previously discussed, the component `ARPlacementWithTouch` enables you to cast a ray into the environment, detecting collisions in real time. By visualizing the collision point, we can accurately determine the exact location to create our virtual object. Consequently, an indicator is necessary to visually represent the collision point.

To achieve this, we just need add an indicator prefab to the blank field “Placement Indicator” of `ARPlacementWithTouch`.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FPmkJcu6KUmG9JpVxm6Hf%2F8.avif?alt=media&#x26;token=beec9cda-d3f1-4b83-a692-431747cdeeec" alt=""><figcaption></figcaption></figure>

* In Project Window, go Shibi→Shibi AR GUI→Assets→AR Placement→Prefabs, find “Indicator Prefab”, hold and drag it to “Indicator Prefab” field of `AR Placement With Touch`.

## Section 5: Introducing Hand-Tracking Capability

To enable hand interaction with the Buddha, our program needs two capabilities:

1. The program should provide the ability to track the position of our hands. We got `HandTracking` component to help us.
2. The virtual object should be able to interact with the a position(in this tutorial, it’s position of our hands).

Let's start with adding `HandTracking` component.

### 5.1 Implementing Hand-Tracking Component

* First, create an empty GameObject (we already have done this before, try doing it again by yourself), and rename it to "Hand Tracking”.
* In the Inspector panel, click "Add Component", and type "Hand Tracking Manager” to search.
* Click on `HandTrackingManager` to add the component to the "Hand Tracking" GameObject.

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FGUej8KvRe6Xu6h6xRmeC%2F9.avif?alt=media&#x26;token=6b642800-83a8-4120-bdb3-f71efc38cae2" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
It's worth noting the configurations within `HandTrackingManager`: we've opted for "Max Hand Count: One" and enabled "Hand Joint Visibility." The former indicates that we are currently tracking joints of one hand(not both), while the latter ensures that the tracked joints are visually displayed for reference.
{% endhint %}

However, this is not enough to enable Hand-Tracking capability since it requires access to the depth map. We need to enable the Lidar camera of our mobile to obtain the depth map.

To enable the Lidar camera, use a device that has a Lidar camera, and manually add `AROcclusionManager` to the main camera.

{% hint style="info" %}
The `AROcclusionManager` is a component for AR applications. It allows you to implement occlusion effects in the AR environment, meaning virtual objects can be occluded by real-world objects, enhancing the interaction and realism between the virtual and real environments.
{% endhint %}

*   Find Object "HoloKit Camera" under Object "XR Origin", click "Add Component" and type: "AR Occlusion Manager", click on `AROcclusionManager` to apply it:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2Fcw3B3Oq9fMYLxx4XwZQo%2Fimage.png?alt=media&#x26;token=1581b1cc-bab7-4864-8925-55294cd2c95a" alt=""><figcaption><p>Add AR Occlusion Manager</p></figcaption></figure>

With the hand tracking feature now in place, if we proceed to build our app for a mobile device, we should observe the following outcomes:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2F3iqFJwa7euvMxlFdVt2a%2F10.avif?alt=media&#x26;token=4fc30ae8-6e04-4588-be34-fd57a3c92804" alt=""><figcaption></figcaption></figure>

### 5.2 Getting a Hand's Representation

As we observe, hand tracking feature aids us in monitoring all the joints of our palm. However, in our scenario, we don't require the entirety of information about each joint. Our objective is to determine a single position that roughly represents the hand's position.

First, let’s choose a Joint to represents the hand's position:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FdzTtcqTg6hX1uJcaREPM%2Fimage.png?alt=media&#x26;token=84b43810-0444-4175-9756-012b94c795ed" alt=""><figcaption><p>All Tracked Joints</p></figcaption></figure>

As the image shows, the joint MiddleMCP seems at the most center part of an opened palm. I’ll take this joint.

* Open the sub-hierarchy of the object `Hand Tracking Manager`, then open the sub-hierarchy of Object Hand 0. You’ll find object MiddleMCP and the other joint objects here.

Remember the location of MiddleMCP, as we'll utilize it later on.

## Section 6: Implementing Scene Manager

In Unity, a scene manager script is commonly used to centrally handle operations such as loading, switching, and unloading game scenes.

By customizing a scene manager script, you can gain more flexibility in controlling scene transitions, data communication between scenes, and handling some global logic, etc…

Today, we're focusing on the feature handling some global logic. With the scene manager we provided, `BuddhaSceneManager` , we'll take object "MiddleMCP" as a public parameter. This approach enables the Prefab Buddha, which will be generated at runtime, to easily locate and utilize its position.

* First , in the Hierarchy Window, right-click, create Empty Object, and rename it to "Scene Manager"
* Next , click “Add Component”, type in "Buddha Scene Manager", click on `BuddhaSceneManager` to apply it:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FQYoJxQm1z4mOEKUpXHyc%2F11.avif?alt=media&#x26;token=7d7c4e48-424c-48b5-a5a6-1d7277455f47" alt=""><figcaption></figcaption></figure>

* To take object "MiddleMCP" as a public parameter. Drag the object “MiddleMCP” onto the blank field of `BuddhaSceneManager`:&#x20;

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2F5QQja5WfCofU0zphE8kC%2F12.avif?alt=media&#x26;token=ec749d64-4393-4675-81f9-e3a63893596b" alt=""><figcaption></figcaption></figure>

## Section 7: Integrating Background Music

In the realm of AR, visual elements often steal the spotlight, but there's an equally vital player that significantly shapes the immersive landscape: sound.

We also want a background music to enhance the whole experience. Let's add an `Audio Source` component to the object Scene Manager.

* Click on object Scene Manager to select it, click “Add Component”, type in “Audio Source”, click `AudioSource` to apply it:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2Fmu96D1bqzmmuvT1uu7lR%2F13.avif?alt=media&#x26;token=1d07b0b5-c1a6-462f-82b8-804bb2acfdd2" alt=""><figcaption></figcaption></figure>

* Click the circle to the right of Audio Clip to select an audio file to play.
* In the search bar above, type "LightTexture," and click audio file "M\_LightTexture" to preview its sound.
* Double-click on this audio file to confirm your selection.

![](https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FGjj4yfezQH1F6UvbTog5%2F14.avif?alt=media\&token=1b92d0b6-d7a0-488c-b763-a9c885507e4b)

We should also ensure to enable the "Loop" option to enable the song to play continuously. This is important because if the user's experience time surpasses the duration of the music, we wouldn't want the scene to become silent, correct?

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FVciB5LWVe9eu3sgKUi8B%2Fimage.png?alt=media&#x26;token=0eb7e277-4e07-4401-a87c-b47792f775fb" alt=""><figcaption><p>Check Loop Option</p></figcaption></figure>

All set! Welcome to a world filled with sound!

## Section 8: Testing in Play Mode

Before build to your mobile, remember testing your app in the Play Mode, to ensure interactions and functionalities work as expected and check for any console errors.

The reasons for doing this are as follows:

1. **Testing in editor mode is quick.** With a single click, we can enter Play Mode, check if there are any errors in the Console Window, and verify if certain processes are meeting our expectations. If issues arise, it's easy to exit Play mode, make adjustments, and enter Play mode again. This allows for rapid iteration of our scene.
2. **Building the mobile app can be slow.** Building onto a mobile device and waiting for the app to open on the device can take several minutes (even longer for the first build or for large projects) before seeing the results.
3. **Multiple testing cycles speed up development.** When developing an MR(AR) application, it's recommended to conduct multiple testing cycles, involving both editor testing and building to the mobile device.

However, since the project of this course is small at size, we can do our first test towards the end. If your project is larger, be sure to conduct multiple tests during your development. Nobody wants to complete the entire application enthusiastically, only to enter Play mode and discover a multitude of error messages, right?

Let's enter Play Mode to check for errors and observe the effects in the Simulator Window:

*   Click “Play” icon to enter Play Mode. Then, click on the screen in the Simulator Window to create the Prefab Buddha:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FcRV7U1J75od06ckKJhJg%2Fimage.png?alt=media&#x26;token=6afc3a96-9727-4a3e-9baf-1e5c067006ad" alt=""><figcaption><p>You should observe a vibrant particle-style Buddha being created.</p></figcaption></figure>
* Try moving the position of the object “MiddleMCP" to get closer to the Buddha and observe the interaction effect:

<figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2F6yNSv54vPM33m5eBvjJS%2F15.avif?alt=media&#x26;token=6ddb18fa-8b1a-4413-b292-ac1720d8757f" alt=""><figcaption></figcaption></figure>

* Click “Play” icon again to exit Play Mode.

{% hint style="info" %}
Please remember that any modifications to objects and object hierarchy within the scene in Play Mode will not be saved.
{% endhint %}

And now we're ready to proceed with building the app and seeing the results on your mobile device!

## Section 9: Building Mobile Applications

1.  Open Unity's top menu and navigate to File → Build Settings. You do not need to change any build settings for this project cause we already done this step for you:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FLnqc3o9cjNEdLrLYRQUr%2Fimage.png?alt=media&#x26;token=fb267360-582a-4b97-a11b-3ba40a1401e0" alt=""><figcaption><p>Open Build Settings</p></figcaption></figure>
2.  Click on "Add Open Scene" to add the current scene to the list for building the app:

    <figure><img src="https://761546874-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FGgGuLHKzjNAAsbVXzyoG%2Fuploads%2FdJSi3svzADtx2KD2pSmf%2Fimage.png?alt=media&#x26;token=80adfeab-4539-4882-af0d-ec5f9019f842" alt=""><figcaption><p>Add Open Scene</p></figcaption></figure>
3. Click on "Build And Run". Ensure that your Xcode configuration is correct and that your iPhone is connected to the computer and is unlocked. Wait for the build to complete.

{% hint style="info" %}
The distinction between "Build" and "Build And Run":

* The "Build" button creates an Xcode project for you. After that, you need to manually open Xcode, load the project based on the file path, and click "Run" to build the mobile app.
* "Build And Run" combines the aforementioned steps, automatically opening Xcode and building the mobile app.
{% endhint %}

After the build completes, the app will automatically open on your phone. Move your phone around, facing different surfaces (floor, walls), and tap the screen to create the Buddha. Try waving your hand near the Buddha to trigger the interactive effect.

## Section 10: Summary: **Embark on Your AR Journey!**

In this course, we've explored how to create awe-inspiring Augmented Reality (AR) applications using Unity and AR Foundation. We began by setting up our project and guided you through essential steps, from window layout and virtual object creation to gesture interactions, sound effects, hand-tracking and HoloKit integration. Along the way, you've learned how to manage scenes and enhance user experiences.

As part of your journey into AR, you've taken a solid step forward. With further exploration and practice, you can apply what you've learned to various scenarios, creating incredible AR experiences that captivate your audience. Thank you for participating in this course, and may you achieve more successes and innovations in the world of AR.

**Keep exploring and pushing boundaries. Best of luck on your journey!**

If you have any further questions, suggestions, or needs, please don't hesitate to reach out. I'm here to assist you

## Further Readings

#### VFX

[Unity Visual Effect Graph](https://unity.com/visual-effect-graph)
