
==== Front
bioRxiv
BIORXIV
bioRxiv
2692-8205
Cold Spring Harbor Laboratory

39229047
10.1101/2024.08.14.607690
preprint
2
Article
An instantaneous voice synthesis neuroprosthesis
Wairagkar Maitreyee http://orcid.org/0000-0002-9546-2843

Card Nicholas S. http://orcid.org/0000-0002-6858-268X

Singer-Clark Tyler http://orcid.org/0000-0001-8636-5075

Hou Xianda http://orcid.org/0009-0002-2066-8561

Iacobacci Carrina http://orcid.org/0000-0003-1690-6272

Hochberg Leigh R. http://orcid.org/0000-0003-0261-2273

Brandman David M. http://orcid.org/0000-0003-3224-7019

Stavisky Sergey D. http://orcid.org/0000-0002-5238-0573

20 9 2024
2024.08.14.607690https://creativecommons.org/licenses/by-nc-nd/4.0/ This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which allows reusers to copy and distribute the material in any medium or format in unadapted form only, for noncommercial purposes only, and only so long as attribution is given to the creator.
http://biorxiv.org/lookup/doi/10.1101/2024.08.14.607690
nihpp-2024.08.14.607690.pdf
Abstract

Brain computer interfaces (BCIs) have the potential to restore communication to people who have lost the ability to speak due to neurological disease or injury. BCIs have been used to translate the neural correlates of attempted speech into text 1–3 . However, text communication fails to capture the nuances of human speech such as prosody, intonation and immediately hearing one’s own voice. Here, we demonstrate a “brain-to-voice” neuroprosthesis that instantaneously synthesizes voice with closed-loop audio feedback by decoding neural activity from 256 microelectrodes implanted into the ventral precentral gyrus of a man with amyotrophic lateral sclerosis and severe dysarthria. We overcame the challenge of lacking ground-truth speech for training the neural decoder and were able to accurately synthesize his voice. Along with phonemic content, we were also able to decode paralinguistic features from intracortical activity, enabling the participant to modulate his BCI-synthesized voice in real-time to change intonation, emphasize words, and sing short melodies. These results demonstrate the feasibility of enabling people with paralysis to speak intelligibly and expressively through a BCI.
==== Body
pmc
