‘Science fiction into science fact’: Decoder can turn your thoughts into readable text

A new technology developed by University of Texas at Austin scientists can turn a person’s thoughts into readable text.

It’s called a semantic decoder, and it uses artificial intelligence to interpret brain activity. The decoder can allow communication with someone who is mentally conscious but may be unable to speak – like someone who’s had a stroke.

Alex Huth is an assistant professor of neuroscience in computer science at UT-Austin. He worked with doctoral student Jerry Tang to develop this decoder. Huth joined Texas Standard to talk about their research, the technology and the legal and ethical concerns surrounding it. Listen to the story above or read the transcript below.

This transcript has been edited lightly for clarity:

Texas Standard: Give us an overview of this brain activity decoder you’ve created. What is it and how does it work exactly? This sounds like one of these almost science fiction devices. 

Alex Huth: Yeah, every day we’re moving a little bit of science fiction into science fact. So the way it works is we put a person in an MRI scanner, kind of like the one where you would get a medical MRI. But we’re doing functional MRI so we can record what’s happening in the brain. We build up a mapping of where different ideas, different thoughts are represented in the brain over many hours of training. And then we can use our algorithms to read those ideas out into words. So we can decode words that somebody is hearing if they’re listening to a podcast, for example, or even just thinking – if they’re telling a story inside their head.

So does the person, the subject, do they have to think particularly hard about it for it to register? How does it work?

So it seems like they have to be very intentionally thinking something. So we were concerned about kind of the mental privacy implications… 

I was going to say, you’ve seen the movie “Minority Report,” right? I would think that there might be something like that.

So we’re worried about this. So we did some tests to see if the person, if they weren’t thinking thoughts very deliberately or if they were trying to think something else, could that upset the decoder? Could that make it not work? And it turns out it can. So the person has to be very deliberately, actively sort of thinking a sentence in their head, and then we can read that out. And of course, this only works on a person where we’ve trained this model with many hours of data.

You mentioned listening to podcasts, for example. This was part of your experimentation? 

Oh, yes.

So is this technology, which uses the kind of artificial intelligence language model that people may know from ChatGPT, how different is it from what has been achieved in the past? 

So there’s a couple strands of work in this direction in the past. One, the most successful so far, has been using electrodes that are actually implanted in the brain. So this is after neurosurgery to implant electrodes, which is only done for people who are undergoing epilepsy surgery or have another medical issue. That’s been very successful, but that gets at a kind of different level of language. So that looks at how a person is trying to move their mouth to speak, for example. Whereas our technology works at this much higher level. It’s about like, what are the ideas the person is thinking about or hearing about? And we can actually read that out, which means that we can do it non-invasively. So we don’t need a person to have brain surgery. They can do it with an MRI, which anybody can do.

How nuanced is it?

It’s okay. So it’s not perfect by any stretch. It doesn’t get the exact words that a person is, you know, hearing or trying to say. It gets the gist. So it gets kind of what are the main ideas. Sometimes it gets phrases correct – you know, specific sequences of words – but it’s still kind of at the level of gist. This is something that we’re working on improving.

So you talked about how this could help people whose brains are active but can’t speak. Do you see any other applications for it?

Yeah. So potentially this could become something that consumers could use at home, certainly not with an MRI scanner, but with other neuro imaging technologies.

I imagine it’s only a matter of time.

Hopefully. It’s kind of hard to say, you know, what kinds of technologies will be available for looking at brain signals in the future, but we’re kind of hopeful that that’s going to work. And if that’s something that people could use at home, then they could potentially use it to interface with their computer. They could use it to search for things – to control their computer without actively, you know, typing things in. We don’t really know what the killer app for that would be yet, but I think there’s a lot of possibilities.

You know, it’s hard not to think about this and think about those who might use it for more nefarious purposes. We’ve heard the expression “thought police.”

Absolutely.

What sort of ethical or legal safeguards are going to have to be built in now that we are, from the sound of it, at that threshold?

This is something we’ve thought about a lot in this work. So we certainly think we’re at the point where considering codified legal safeguards is important – is something that we should be actively worrying about – that people should not have this done without their consent. We think this should not be used for law enforcement purposes because it’s certainly not accurate enough for that. That hasn’t stopped law enforcement in the past, for example, with the polygraph. We don’t want to have that kind of situation repeated. We did some testing on kind of these privacy concerns to see how much control does the person have and that’s where we found, again, that if the person is trying to resist it – trying to make it not work – then they can do that. They can shut it off. And also, we had this lengthy training procedure where the person has to lay in the scanner for many, many hours and listen to podcasts. We tried to see, can we avoid that? Can we take the model that we trained on one person’s brain and apply it to a new person so we wouldn’t need to do this training on them? And that doesn’t work at all, currently. So right now I think the concerns are not very acute, but it’s on the horizon, which is why I think we should think about it.

What’s the next frontier as you see it? 

The biggest kind of avenue forward that we see with this is kind of bigger and better models – like bigger and better A.I. models. As these models get more effective, as we move from the kinds of things that we used in this work – which was GPT-1 from 2019 – now there’s much more effective models out there, and we think that those things will very much improve the system – make it more accurate, which which we’re excited about. But a lot of what we’re doing is really, you know, what we try to study in my lab is how our brains process language, how we understand language. That’s our scientific goal. So a lot of what we’re doing is taking these same kinds of models and using them to try to figure out what the brain is doing, to try to say what are the different parts of the brain responding to, how is language and meaning…

More broadly beyond the technology, in other words.

Right. So what are the scientific implications? So really most of our work is in that direction.

If you found the reporting above valuable, please consider making a donation to support it here. Your gift helps pay for everything you find on texasstandard.org and KUT.org. Thanks for donating today.