Machine Learning Helps Master Imaginary Keyboards

Cloud of keystrokes for imaginary keys.
Cloud of keystrokes for imaginary keys.

Korean engineers developed and tested the effectiveness of the imaginary keyboards for touchscreens – the users participating in the study did not receive any restrictions on working with the keyboard and placed their hands anywhere on the touch panel using imaginary keys in the QWERTY layoutRecognition accuracy of text typed on an invisible keyboard exceeded 95 percent, and typing speed exceeded 45 words per minute.

Many modern devices lack hardware keyboards – in the vast majority of cases, tablets, smartphones and terminals use touchscreens. However, this approach often leads to a decrease in typing speed – when using hardware keyboards, a person feels a keystroke, so he can type quickly using the blind method. On-screen keyboards cannot give such feedback (the maximum is vibration to indicate registration of pressing), therefore, users have significantly reduced typing speed and reduced the number of fingers used to print. As an alternative to character-by-character typing, various gesture input methods are offered (e.g. Swype) however, they also make mistakes, including because they rely heavily on the prediction of the typed text. In addition, this does not solve the problem of blind typing – the user still has to look at the keyboard.

Ue-Hwan Kim and his colleagues at the Korea Institute of Advanced Technology (KAIST) suggested using an imaginary, invisible keyboards for touch typing on touch surfaces that is not limited to a fixed key layout. Engineers have developed a decoder algorithm that, with the help of deep learning and long-term short-term memory implemented by controlled recursive blocks, recognizes characters typed by a person from pressing sequences. The decoder remembers as a context a certain amount of previous data typed by the user, which improves the accuracy of input recognition.

At first, the researchers needed to collect the initial data set for training the algorithm, so they invited 43 volunteers (11 women and 32 men aged 22 to 32 years) to type text on the stand, which consisted of two screens – one was used to display the text, and the second (touch) played the role of a keyboard, only two buttons were displayed on it: “Delete” (to start typing a fragment of text again) and “Continue” (to move to the next fragment). More than a screen with a touchscreen did not display anything, engineers suggested that volunteers simply type in text, like on a regular on-screen keyboard. After warming up 20 sentences, participants were given 150–160 sentences for typing (without rarely used characters, only Latin, spaces, period, apostrophe and comma). As a result, the authors collected 7,245 phrases, which corresponded to 196,194 clicks on imaginary keys. At the same time, the collected point clouds on average really corresponded to the standard QWERTY layout, which confirmed the authors’ assumption that users can really type blindly even without tactile feedback.

The collected data was divided into three parts: training, test and control. To train the algorithm, we used data obtained after typing by two volunteers, and as a control – one participant. The rest of the data was used to train the algorithm. The control dataset was needed to prevent retraining – when the recognition accuracy of the decoder began to decline, the learning process was stopped. The maximum accuracy of the decoder reached 95.84 percent. 

To test the method, the authors recruited a new group of 13 volunteers (8 women, 5 men), confidently owning blind print. Engineers instructed participants to type as quickly and accurately as possible, and gave them to dial both the familiar hardware keyboard and the invisible keyboard on the touchscreen (in random order for each subject). To warm them up, they were asked to type 10 phrases, and then they were given 20 phrases from a previously collected large dataset for typing. As a result, the typing speed for the hardware keyboard was 51.35 words per minute, and for the imaginary keyboard – 45.57 words per minute. In addition, in the experiment, the accuracy of the decoder was slightly higher than when checking on the control data set, and amounted to 96.12 percent.

After completing the experiment, users were also asked to subjectively evaluate the work on an imaginary keyboard. The experiment participants liked that typing did not require any retraining, they also noted that typing in any convenient position of the hands on the panel was convenient. Among the shortcomings, volunteers indicated the inability to press a key with a fingernail, as well as the risk of becoming entangled in closely spaced characters.

The authors separately note that limiting the typing speed on an invisible keyboard may also be due to the choice of a too slow touchscreen for the experiment – some volunteers complained about missed keystrokes. Researchers believe that technology can be improved in the future. The authors are also sure that their development is well suited for virtual reality.

For virtual reality, there are other solutions, including using the familiar hardware keyboard. Logitech, for example, has developed a system for integrating a physical keyboard into virtual reality based on the HTC Vive virtual reality helmet. To do this, a special controller is fixed on the keyboard, which allows you to accurately track the position of the keyboard relative to the helmet, and in virtual reality, the keyboard model and the user’s hands are displayed.

SHARE
Previous articleWindows 10 20H1, build 18950 and Cloud Download
Next articleSay goodbye to Microsoft’s MSDN magazine in November
Jiya Saini
Jiya Saini is a Journalist and Writer at Revyuh.com. She has been working with us since January 2018. After studying at Jamia Millia University, she is fascinated by smart lifestyle and smart living. She covers technology, games, sports and smart living, as well as good experience in press relations. She is also a freelance trainer for macOS and iOS, and In the past, she has worked with various online news magazines in India and Singapore. Email: jiya (at) revyuh (dot) com

LEAVE A REPLY