Developers from the U.S. and China have created a smartphone-based augmented reality system that allows the user to interact with virtual objects directly with the help of hands, not a touchscreen. The development was presented at the UIST 2019 conference.

Augmented reality allows you to embed virtual objects in the real world using the screen and camera of a smartphone or special AR-glasses. Developers often use this in games, but there are also non-game applications of augmented reality. One of the key problems in this area is that in the case of smartphones, the user can interact with virtual objects only through the touch screen, which reduces the sense of the reality of such objects.

A team of engineers led by Jeff Huang of Brown University supplemented the smartphone with a depth camera, which made it possible to track hand movements and use them to interact with virtual objects. An algorithm that can accurately track hand movements for smartphones was created only recently, so the developers used a separate Leap Motion device. It consists of two cameras and three infrared LEDs, thanks to which the device with high accuracy monitors the movements of the fingers.

Since the sensor is designed to work with a computer, not a smartphone, engineers had to add a small computer to the device. The smartphone application runs on the basis of the ARCore framework for Android. The developers taught the system to recognize several gestures by data from a depth camera using the support vector method. In addition to the screen, the system also displays information through sound and vibration.

The authors tested the system on two tasks: the manipulation of household objects and the construction of a small house. A study of 12 volunteers found that the new system allows users to interact more comfortably and efficiently with virtual objects than using only a touchscreen.

There are other projects in which users interact with augmented reality directly with their hands. For example, last year Leap Motion revealed the concept of virtual wearable devices that can be interacted with your fingers in the same way as with real devices by clicking on the buttons and moving them.