Advanced Use of Hand Tracking

Some common use cases

*For guidance on the latest version, refer to the GitHub page.

In the previous section, we demonstrated how to add the HoloKitHandTracker prefab into the scene and successfully tracked user's hands. However, we haven't done any coding yet such as turning hand tracker on and off at runtime and getting the position of a specific landmark position. In this section we will show some common use cases of the HoloKitHandTracker. Same as before, you can directly import this sample through the Unity Package Manager.

The rest of this section will be a step by step guide on how to implement the advanced hand tracking sample. We will start on top of the complete version of the basic hand tracking tutorial project.

This should be the beginning point of this tutorial

Prefab HoloKitHandTracker Configuration

We first have a look at the inspector of HoloKitHandTracker prefab.

The inspector of HoloKitHandTracker prefab

As you can see, there are 3 public properties.

Property Enabled indicates whether we want to turn on the hand tracker. If we set Enabled to true, the hand tracker will be turned on and trying to detect user's hands at every frame. If we set Enabled to false, no hand will be detected. We can dynamically set this property at runtime to save some computational power because the hand tracking algorithm is very computionally expensive.

// Turn on the hand tracker at runtime
// Notice that class HoloKitHandTracker is a singleton
HoloKitHandTracker.Instance.Enabled = true;

// Turn off the hand tracker at runtime
HoloKitHandTracker.Instance.Enabled = false;

Property MaxHandCount indicates the maximum number of hands to be tracked by the algorithm. There are only two options for this property, which is MaxHandCount.OneHand and MaxHandCount.BothHands. This setting is up to your app's need. When you only need one hand to be detected at a time, you can set this property to MaxHandCount.OneHand to save computational power.

// We can also set MaxHandCount at runtime
HoloKitHandTracker.Instance.MaxHandCount = MaxHandCount.OneHand;
HoloKitHandTracker.Instance.MaxHandCount = MaxHandCount.BothHands;

Property DebugMode controls the visibility of detected hands. If we set DebugMode to true, when a hand is detected, there are 21 landmarks with different colors indicating the position of each hand joint. However, in a real HoloKit app, we may not want to show those colorful landmarks. We can set DebugMode to false to make those landmarks invisible.

// We can also set DebugMode at runtime
HoloKitHandTracker.Instance.DebugMode = true;
HoloKitHandTracker.Instance.DebugMode = false;

Get The Position of a Specific Landmark

The most common use case of HoloKitHandTracker might be making a game object interacting with the position of a specific hand landmark, such as the point of the index finger.

In this sample project, we will attach a sphere onto the user's index finger. We first add a sphere into the scene and create a new script called IndexFingerSphere. We set the size of the sphere to 0.1 and attach the script onto it. The code of the script is shown below. Basically, in every frame, we first check whether there is at least one hand detected. If there is one, we get the first detected hand (there could be two) and set the sphere to the position of its index finger.

public class IndexFingerSphere : MonoBehaviour
{
    private HoloKitHandTracker _handTracker;

    private void Start()
    {
        // Get the reference of the hand tracker singleton instance
        _handTracker = HoloKitHandTracker.Instance;
    }

    private void Update()
    {
        // Check whether there is at least one hand detected in the current frame
        if (_handTracker.AvailableHandCount > 0)
        {
            // Get the first detected hand
            HoloKitHand hand = _handTracker.Hands[0];
            // Attach the sphere to the end of the index finger
            transform.position = hand.GetLandmarkPosition(LandmarkType.Index3);
        }
        else
        {
            // There is no hand detected in this frame
            // Move the sphere to the sky so the user cannot see it
            transform.position = new Vector3(0f, 99f, 0f);
        }
    }
}

Please notice that, although the hand tracking algorithm can detect user's both hands, it cannot tell the handedness. It doesn't know which hand is left and which hand is right. The index of the list property HoloKitHandTracker.Hands only indicates the detection order, where HoloKitHandTracker.Hands[0] is the first hand being detected in the frame, and HoloKitHandTracker.Hands[1] is the second one.

// When there is only one hand detected in the current frame
if (HoloKitHandTracker.Instance.AvailableHandCount == 1) {
    if (HoloKitHandTracker.Instance.Hands[0] != null) {
        // This if statement is true
        // The first hand element has value
    }
    
    if (HoloKitHandTracker.Instance.Hands[1] == null) {
        // This if statement is also true
        // The second hand element is null
    }
}

You might have noticed in the current project, there in only one IndexFingerSphere object. When two hands are detected, the IndexFingerSphere will jump between the two index fingers. Let's change the IndexFingerSphere script and add another sphere object to fit this need.

public class IndexFingerSphere : MonoBehaviour
{
    // Indicate which hand this sphere is attached to
    // This should be 0 for one sphere and 1 for the other sphere
    public int HandIndex = 0;

    private HoloKitHandTracker _handTracker;

    private void Start()
    {
        // Get the reference of the hand tracker singleton instance
        _handTracker = HoloKitHandTracker.Instance;
    }

    private void Update()
    {
        // Check whether the desired hand is detected in the current frame
        if (_handTracker.AvailableHandCount > HandIndex)
        {
            // Get the desired hand
            HoloKitHand hand = _handTracker.Hands[HandIndex];
            // Attach the sphere to the end of the index finger
            transform.position = hand.GetLandmarkPosition(LandmarkType.Index3);
        }
        else
        {
            // There is no hand detected in this frame
            // Move the sphere to the sky so the user cannot see it
            transform.position = new Vector3(0f, 99f, 0f);
        }
    }
}

Now we have two IndexFingerSphere objects, one for each hand

Now when two hands are detected at the same time, there will be two spheres attached to each index finger respectively.