Researchers ‘decode’ Mandarin Chinese from neural signals

[aum]

Advances in brain-computer interfaces could help millions who use tonal languages speak again after stroke or disease

< View the video at the source page>

 

Researchers used a brain implant to decode the Mandarin sentence “Happy New Year 2025” from a woman’s brain activity while she spoke aloud.Youkun Qian

 

Brain-computer interfaces (BCIs) can restore communication for people who have lost their speech to a stroke or neurological disease. But these experimental systems, which combine sensors placed in the brain with algorithms that turn the neural signals of attempted speech into words, have so far been developed mostly for English speakers.

 

BCIs for Mandarin Chinese, a tonal language in which stark changes in pitch distinguish the meanings of words, has lagged. Today in Science Advances, a group of researchers in Shanghai reports success in “decoding” Mandarin speech from the neural signals of a healthy person as she spoke—and, in a first, generated Chinese text onscreen in real time.

 

The progress “opens [BCIs] up to huge populations of potential patients,” says Matthew Leonard, a neurolinguistics expert at the University of California (UC) San Francisco who was not involved in the study. There are more tonal languages in the world than nontonal languages, he notes. And the new work marks only one of several recent strides toward Mandarin speech BCIs.

 

Some English-language BCIs have already generated clear audio, nearly instantaneously, from the brains of people who struggle to speak. Interpreting precise tonality from brain signals adds an extra layer of complexity to designing speech BCIs for Chinese, though Leonard notes prior research has shown there are more similarities than differences in how the brain processes tonal and nontonal languages.

 

The Shanghai team—led by neurosurgeon Jinsong Wu of Fudan University and neural engineering scientist Zhitao Zhou at the Shanghai Key Laboratory of Clinical and Translational Brain-Computer Interface Research—has been working on a Mandarin decoder for years. Wu’s clinical interest in rehabilitating patients with speech loss after brain surgery helped drive the research.

 

In work published in 2023, the group recorded brain activity from people speaking Mandarin during surgeries for brain tumors.

 

Because the brain is insensitive to pain, the patients could be kept awake for their procedures, during which electrode sensors were used to map brain activity, including from the regions involved in speech production. The researchers found that the data these electrodes recorded could be decoded to produce some Mandarin syllables, including tones.

 

Now, they have done a much more extensive BCI experiment. The study, conducted with the Chinese BCI company NeuroXess, involved a single participant, a 43-year-old woman with epilepsy whose speech was normal. The researchers took advantage of electrodes that had been temporarily implanted to allow mapping of her brain for a surgery to treat her epilepsy. Over nearly 2 weeks while she was hospitalized, the investigators recorded her brain activity as she repeated, in session after session, about 400 of the tonal syllables that comprise the phonetic building blocks of Mandarin. They used those data to train the BCI system.

 

Then they turned brain activity back into words. In December 2024, the patient was able to convey brief New Year’s greetings in real time on a computer screen. Seeing the right Chinese characters pop up as she spoke was “incredibly exciting,” Wu says.

 

The team did not try to decode the patient’s silent or imagined speech. Rather, she read onscreen prompts aloud, and the researchers relied on the onset of those speech sounds to indicate where to start decoding her neural signals. The communication was slow—equivalent to 50 Chinese characters per minute, or one-fifth the speed of normal speech—though about 70% accurate.

 

“There’s a large jump [in] going from healthy speech subjects to a medical BCI for someone who cannot talk,” says BCI researcher Sergey Stavisky of UC Davis. And studying a person with healthy speech results in cleaner training data than working with people whose speech is impaired. Still, generating Chinese characters “online” during speech—not just by reanalyzing recorded neural data later on—is “genuine progress,” he says.

 

Other teams have also reported progress toward a Chinese speech prosthesis. In April, BCI researcher Jie Yang and his colleagues at Westlake University reported results from four patients with epilepsy who, like the patient in the Shanghai study, modeled hundreds of syllables over days or weeks as they awaited their surgeries with electrodes on their brains. That group’s offline decoder later chose the participants’ intended words with 70% accuracy, and one-third of full sentences came out without errors. The Fudan group’s “overall experimental pipeline and underlying idea are quite similar to ours,” Yang says. Both studies argue for the value of “robust syllable-level mappings” as the basis for decoding Mandarin.

 

Wu says his team is working to make the decoding faster and more accurate. The researchers now want to work with Chinese patients who have speech difficulties caused by stroke or the neurodegenerative disease amyotrophic lateral sclerosis. They are also developing a wireless, implantable system that could allow for long-term use in such patients. The more data they can collect, the better they hope their high-tech Mandarin interpreter will sound.

 

Source

Edited November 7, 2025 by aum