Facebook has released an update for the Oculus Quest standalone VR helmet, which allows it to track the user’s hand movements using cameras without using controllers.

In general, helmets of virtual reality are used not for watching movies or videos, but for active interaction with the virtual world in games or educational applications. For this, standard controllers are usually used to track the movements of the arm as a whole, or of individual fingers. However, this is not as convenient as interacting in the usual way with free hands. To do this, some manufacturers created external motion capture sensors, but this “binds” the user to the same room with the sensors or requires fixing the sensor to the helmet itself.

In 2018, Oculus developers introduced the new Quest VR helmet, one of the features of which, in addition to battery life, was four cameras on the front surface, thanks to which the device can track its movement relative to the room. Now developers have begun to use cameras to create a dynamic model of hands. They note that the feature was only added now because they were able to create fairly effective machine learning algorithms and create a brush model for each hand.

At the initial stage, hand tracking is available only in Oculus own applications, but next week third-party developers will get access to the SDK to work with the recognition function and will be able to integrate it into their applications.

In addition to interacting with a virtual environment with free hands in the field of virtual reality, there is another, albeit related problem. It lies in the difficulty of reconstructing physical contact with virtual objects. Engineers have developed many solutions to this problem, but so far they are almost never used in serial devices.