New computer interface translates mental handwriting into text

Using a sensor implanted on the brain, a team of neuroscientists developed an interface that decodes neural signals associated with handwriting and transcribes it as text in real-time, pioneering one of the very first brain-to-text communication systems via thought-projected writing.

A study published by the journal Nature last May 12, 2021 presented a proof of concept which enabled researchers to decode speech from brain activities associated with areas for hand control. This study is one of the collaborative researches from a long-term clinical trial called BrainGate.

The patterns of electrical activity as interpreted by the algorithm which trace a path of thought projected handwriting. Photo taken from Nature’s “High-performance brain-to-text communication via handwriting” paper.

In this investigation, a participant with paralysis used this system to type words on a computer just by merely thinking. During his sessions, a brain-computer interface (BCI) recorded his brain activity as he imagined handwriting the sentences shown to him on screen. Then, an algorithm picked up the patterns produced by his brain and translated them as text in real-time.

By imagining how to write, the clinical trial participant was able to type with 94 percent accuracy at a rate of 90 characters per minute, more than double the speed of previous BCIs.

Previous Brain-Computer Interface Systems

Experimental set-up of the point-and-click navigation system. Photo taken from eLife’s “High-performance communication by people with paralysis using an intracortical brain-computer interface” paper.

In 2017, BrainGate researchers developed a similar BCI to elicit high-performance communication by people with paralysis. The difference, however, is that the past system made use of a point-and-click navigation system to type characters using the brain. The concept of this interface is to let the person communicate with a thought-controlled cursor, pushing the keys of the characters displayed on a keyboard through the brain.

More recently, in April 2021, the BrainGate collaborative completed their first human trial on the same system but with high-bandwidth wireless BCI.

Still, compared to the point-and-click system, the handwriting system functioned faster and with more efficiency.

Feasibility for Clinical Trials

Although that is the case, the research team necessitates several improvements before making the BCI available for clinical use. Among these developments are making the sensor implant smaller and capable of wireless transition, recalibrating the neural activity to account for variables, and improving the algorithm with machine learning.

Moreover, the research team also intends to find new, speech-centered ways to enable faster communication than with handwriting.

This study is the latest in a series of BCI advancements made by BrainGate collaborative, specifically written by researchers Francis R. Willett, Donald T. Avasino, Leigh R. Hochberg, Jaimie M. Henderson, and Krishna V. Shenoy.

---

References

Willett, F.R. et al. (2021). High-performance brain-to-text communication via handwriting. Nature 593, 249–254. https://doi.org/10.1038/s41586-021-03506-2

Pandarinath, C. et al. (2017). High-performance communication by people with paralysis using an intracortical brain-computer interface. eLife, 6. doi:10.7554/elife.18554