When playing video on a projector, the light from it falls on the walls and creates changes in brightness and shadow on them. American developers have created an algorithm that recreates the original video using these lighting effects.

Researchers have long suggested using algorithms that restore the image in the absence of line of sight. Potential applications for this are driverless cars, which using this technique can determine that an object is moving around the corner, as well as reconnaissance operations, within which it is possible to intercept some data in this way.

However, these algorithms often have serious limitations, which make them difficult to put into practice. For example, they usually require careful calibration for each scene to restore the image with relatively high accuracy. Fredo Durand and his colleagues from the Massachusetts Institute of Technology have created an algorithm that allows you to not calibrate and at the same time receive a sufficiently high-quality (in comparison with similar algorithms) video clip from the video recording changes in light and colour at other objects.

The researchers decided to look at this problem in the form of matrix decomposition. In this case, the image observed on the wall or other surface changes in light is formed from the original image on the projector or monitor, as well as the matrix, which is responsible for the transfer between the two images. This matrix depends on the physical parameters of the environment: the screen on which the original movie is displayed, the wall on which the reflection falls, objects that give shadows, and others.

Based on this view of the transition problem between primary and secondary image, the researchers created an algorithm that recreates both unknown matrixes that correspond to the formation of the observed secondary image. It consists of two reconciliation neural networks, each of which forms its own matrix: a transition matrix or a matrix corresponding to the original image. The authors used an approach previously proposed by a Russian-British team of researchers. Briefly, it is that for some tasks related to the image (such as noise suppression or increased resolution), the neural network with accidentally initiated parameters can produce excellent results without training on datasets.

The results of the two neural networks are combined and then compared with a matrix formed from the observed secondary image. In the process of training, neural networks adjust their parameters and gradually the difference decreases. This allows you to train the neural network responsible for restoring the original image.

In the video published by the researchers, you can see that the neural network copes well with simple situations, for example, when a person juggles big balls. If it collides with colour rollers that contain more details, the results are less impressive, but even then the algorithm as a whole is doing its job.