Engineers from Sony have created a prototype system that allows teachers and experienced musicians to directly transmit finger movements while playing the piano to a student. The video describing the system is published on Sony’s YouTube channel.

When learning to play the majority of musical instruments, students can reproduce the technique of their teacher’s performance based only on their vision, hearing and a general understanding of the principles of playing the instrument. In addition, usually during the performance, the musician himself usually cannot fully control every aspect of the movements of the fingers and thereby analyze his mistakes or adjust the style.

A team of Japanese engineers from Sony, led by Shinichi Furuya, have created a system that allows direct movement between the student and the teacher. This group of researchers has been studying aspects of piano playing through technical means for some time. For example, they recently found out that with the help of an exoskeleton glove that a long practice of playing the piano enhances the coordination of the somatosensory and motor systems.

Now they have created a connected system consisting of a modified piano, play-recording gloves and exoskeleton gloves to play recorded parts. In the piano under a row of keys, engineers have installed optical sensors capable of measuring the force and speed of pressing. As a brush exoskeleton, engineers have taken a ready-made device from exiii Inc, capable of bending each finger to the right angle. The pianist-teacher wears a lighter glove, which only measures the angles of the bend of the fingers.

All this data is collected into a computer that processes data and transmits a signal from the teacher’s glove to the student’s exoskeleton glove. Engineers have shown that the system can work in real-time by transferring finger movements between two people, as well as reproducing recorded sequences, for example, to analyze your own movements after the play. The system is also able to visualize the performance on a monitor next to the piano in real-time.