AI Can Now Decode Words Directly from Brain Waves
are teaching computers to read words straight out of people’s brains.
Servick, writing for Science, reported
this week on three papers posted to the preprint server bioRxiv in which three
different teams of researchers demonstrated that they could decode speech from
recordings of neurons firing. In each study, electrodes placed directly on the
brain recorded neural activity while brain-surgery patients listened to speech
or read words out loud. Then, researchers tried to figure out what the patients
were hearing or saying. In each case, researchers were able to convert the
brain’s electrical activity into at least somewhat-intelligible sound files.
The first paper,
posted to bioRxiv on Oct. 10, 2018, describes an experiment in which
researchers played recordings of speech to patients with epilepsy
who were in the middle of brain surgery. (The neural recordings taken in the
experiment had to be very detailed to be interpreted. And that level of detail
is available only during the rare circumstances when a brain is exposed to the
air and electrodes are placed on it directly, such as in brain surgery.) [3D Images:
Exploring the Human Brain]
patients listened to the sound files, the researchers recorded neurons firing
in the parts of the patients’ brains that process sound. The scientists tried a
number of different methods for turning that neuronal firing data into speech
and found that “deep
learning” — in which a computer tries to solve a problem more or less
unsupervised — worked best. When they played the results through a vocoder,
which synthesizes human voices, for a group of 11 listeners, those individuals
were able to correctly interpret the words 75 percent of the time.
can listen to audio from this experiment here.
The second paper,
posted Nov. 27, 2018, relied on neural recordings from people undergoing
surgery to remove brain tumors. As the patients read single-syllable words out
loud, the researchers recorded both the sounds coming out of the participants’
mouths and the neurons firing in the speech-producing
regions of their brains. Instead of training computers deeply on each
patient, these researchers taught an artificial neural network to convert the
neural recordings into audio, showing that the results were at least reasonably
intelligible and similar to the recordings made by the microphones. (The audio
from this experiment is here
but has to be downloaded as a zip file.)
The third paper,
posted Aug. 9, 2018, relied on recording the part
of the brain that converts specific words that a person decides to speak
into muscle movements. While no recording from this experiment is available
online, the researchers reported that they were able to reconstruct entire
sentences (also recorded during brain surgery on patients with epilepsy) and
that people who listened to the sentences were able to correctly interpret them
on a multiple choice test (out of 10 choices) 83 percent of the time. That
experiment’s method relied on identifying the patterns involved in producing
individual syllables, rather than whole words.
goal in all of these experiments is to one day make it possible for people
who’ve lost the ability to speak (due to amyotrophic
lateral sclerosis or similar conditions) to speak through a computer-to-brain
interface. However, the science for that application isn’t there yet.
the neural patterns of a person just imagining speech is more complicated than
interpreting the patterns of someone listening to or producing speech, Science
reported. (However, the authors of the second paper said that interpreting the
brain activity of someone imagining speech may be possible.)
also important to keep in mind that these are small studies. The first paper
relied on data taken from just five patients, while the second looked at six
patients and the third only three. And none of the neural recordings lasted
more than an hour.
the science is moving forward, and artificial-speech devices hooked up directly
to the brain seem like a real possibility at some point down the road.