A new mind-reading device, developed by researchers at the University of Texas, can decode the content of your thoughts.

Researchers at the University of Texas at Austin have developed a groundbreaking artificial intelligence system known as a semantic decoder. This remarkable system can translate a person’s brain activity, whether they are listening to a story or silently imagining telling a story, into a continuous stream of text.

The study, led by Jerry Tang, a doctoral student in computer science, and Alex Huth, an assistant professor of neuroscience and computer science, has been published in the prestigious journal Nature Neuroscience.

What sets this AI system apart from others in development is its noninvasive nature, eliminating the need for surgical implants. Participants are not limited to using a predetermined list of words either.

The process involves training the decoder by having individuals listen to hours of podcasts in an fMRI scanner. Subsequently, when the participant is receptive to having their thoughts decoded, simply listening to a new story or envisioning telling a story prompts the machine to generate corresponding text based solely on brain activity.

“For a noninvasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” Huth added. “We’re getting the model to decode continuous language for extended periods of time with complicated ideas.”

The outcome is not an exact transcript, but rather a designed representation capturing the essence of the spoken or thought content, although not flawlessly. In approximately half of the instances, when the decoder was trained to monitor an individual’s brain activity, the system generated text that closely aligned with, and sometimes accurately mirrored, the intended meanings of the original words.

For instance, during experiments, a participant who heard the phrase, “I don’t have my driver’s license yet,” had their thoughts translated as, “She has not even started learning to drive yet.” Similarly, when exposed to the statement, “I didn’t know whether to scream, cry, or run away. Instead, I said, ‘Leave me alone!'” the decoded output was, “Started to scream and cry, and then she simply stated, ‘I told you to leave me alone.'”

Starting from a previous iteration available as an online preprint, the scientists proactively tackled concerns surrounding the potential misuse of their technology. The research paper explicitly outlines that successful decoding was achieved solely with the collaboration of willing participants who actively engaged in the decoder training process. Unsurprisingly, attempts to decode brain activity from individuals who had not undergone the decoder training rendered unintelligible results. Furthermore, if participants who had been trained on the decoder later exhibited resistance, such as thinking unrelated thoughts, the results became similarly impractical and unusable.

“We take very seriously the concerns that it could be used for bad purposes and have worked to avoid that,” Tang added. “We want to make sure people only use these types of technologies when they want to and that it helps them.”

The researchers not only asked participants to listen or think about stories but also to watch four short, silent videos while in the scanner. Through analyzing the subjects’ brain activity, the semantic decoder was able to accurately describe certain events from the videos.

Currently, the system is not feasible for use outside of the laboratory as it requires an fMRI machine, which can be time-consuming. However, the researchers anticipate that this technology could be adapted to other, more portable brain-imaging systems, such as functional near-infrared spectroscopy (fNIRS).

“fNIRS measures where there’s more or less blood flow in the brain at different points in time, which, it turns out, is exactly the same kind of signal that fMRI is measuring,” Huth adds. “So, our exact kind of approach should translate to fNIRS,” remarked Huth, highlighting that there may be a compromise in resolution when using fNIRS.

Source: 10.1038/s41593-023-01304-9 

Image Credit: Nolan Zunk/University of Texas at Austin