New AI Composes Songs From Silent Performance Videos

12:13 minutes

There have been many awkward attempts in the quest to train algorithms to do what humans can. Music is a prime example. It turns out that the process of turning the individual notes of a composed piece into a fully expressive performance—complete with changes in loudness and mood—is not easy to automate. 

But a team at the University of Washington has been closing in on a way to get close, in research they presented at a machine learning conference late last year. Their AI tool called “Audeo,” combining the words “audio” and “video,” watches a silent video of a piano performance, like this one by virtuoso Paul Barton, performing Piotr Ilyich Tchaikovsky’s Valse Sentimentale (Op. 51).

Then, using only the visual information, Audeo produces music with the expressiveness and interpretative idiosyncrasies of the musician it just watched. 

Depending on the kind of synthesizer software the researchers partnered with Audeo, the result can be very expressive, but slightly “noisy”:

Credit: University of Washington

Producer Christie Taylor talks to lead author Eli Shlizerman about how one trains an algorithm to make art, and how such tools could help make music both more accessible, and easier to engage with.

Further Reading

Donate To Science Friday

Invest in quality science journalism by making a donation to Science Friday.


Segment Guests

Eli Shlizerman

Eli Shlizerman is an assistant professor of applied mathematics and electrical and computer engineering at the University of Washington in Seattle, Washington.

Segment Transcript

IRA FLATOW: This is Science Friday, I’m Ira Flatow. Alzheimer’s disease is still such a mystery. There are still no known ways to stop this degenerative disease. Available drugs, well, they work only to slow symptoms. But there is a potentially fruitful area. Researchers have been investigating the connection between type 2 diabetes, insulin resistance, and Alzheimer’s for decades. It’s known, for example, that having diabetes elevates your risk of Alzheimer’s. The question is, why?

There is an intriguing clue. A team at Brigham Young University and Washington University in St. Louis found that brain cells in regions most affected by Alzheimer’s seem to show impaired ability to metabolize sugars. Here’s lead author, Dr. Benjamin Bikman.

BENJAMIN BIKMAN: We identified every relevant gene involved in glucose metabolism and all of the different cells of the hippocampus. And every gene in every cell was compromised or significantly down.

IRA FLATOW: This finding supports characterizing Alzheimer’s as a metabolic disorder, suggesting that lifestyle or dietary changes may help prevent or treat Alzheimer’s.

BENJAMIN BIKMAN: As the brain is becoming increasingly insulin resistant, it’s becoming increasingly less able to get adequate glucose. So you end up in this sort of tragic scenario where the body is flushed with glucose but the brain can’t use it.

IRA FLATOW: Dr. Benjamin Bikman. Here to update and talk more about the research into Alzheimer’s and blood sugar is Dr. Shannon Macauley, assistant professor of geriatric medicine. She studies the connection between Alzheimer’s disease and type 2 diabetes at Wake Forrest school of Medicine in Winston-Salem, North Carolina. And I should note, Dr. McCauley is not affiliated with the new research I just mentioned. Welcome back, Shannon.

SHANNON MACAULEY: Thank you, it’s a pleasure to be here.

IRA FLATOW: First of all, what did this team look at in this new research? Can you fill us in on that.

SHANNON MACAULEY: Absolutely. So this team took post-mortem brain samples and looked at how the genes change that really regulate brain metabolism in a few key regions in the brain. And I think what they found is that glucose, which is a fuel for our brain, was unbelievably impaired in the regions that are prone to Alzheimer’s pathology such as amyloid plaques and neurofibrillary tangles.

And what they also found is it wasn’t just the neurons getting sick. In fact, they found that oligodendrocytes, which are myelinating cells of the brain, were the ones that seemed to be unbelievably impaired using their approach. So one, it really reinforces the idea that as we develop Alzheimer’s disease that metabolic dysfunction could be playing a causal role in this disease, but also that cell types in these brain regions vulnerable to Alzheimer’s pathology are impacted.

IRA FLATOW: Why couldn’t those brain cells get the glucose they need?

SHANNON MACAULEY: So it’s hard to know, from this study and a rich literature over the last 20 years, whether it’s the transport of glucose into the brain that’s impaired or whether there’s an intrinsic deficit in these different cell types to use glucose properly. What we do know is that the regions that are prone to Alzheimer’s disease have a really high reliance on glucose for fuel. They’re very active regions, they’re well-connected regions, and those network failures are typical of Alzheimer’s disease. And so what this study really showed was that they do not process glucose correctly. And how much of that is a cause or an effect still needs to be parsed out with future studies.

IRA FLATOW: And as someone who’s looking for this connection between Alzheimer’s and insulin resistance, what excites you about a study like this and why?

SHANNON MACAULEY: For a long time, individuals just thought that the misuse of fuel, or the hypometabolism we see in Alzheimer’s, which is a key feature that most people use with neuroimaging, was just because the neurons were dying or the neurons were dysfunctional. And what really excites me about this study is it’s really showing that perhaps we don’t need to focus as much on the neurons, but we need to look at the glia. The glia are the cells that support the neurons, but they’re way more than that. So the oligodendrocytes, our myelinating cells, the astrocytes in the microglia have a very complex role in inflammation and just normal brain function. And I think what’s really rich about this study is it shows that these other cell types are contributing actively to the energy deficit we see in Alzheimer’s disease.

IRA FLATOW: You know, I’ve heard years ago people would talk about Alzheimer’s as possibly being, quote, “type 3 diabetes” because of this insulin resistance, and the same sort of jargon gets thrown around with Alzheimer’s this way. Could there be a connection like that?

SHANNON MACAULEY: Absolutely, and I think there’s a couple of things that happen with type 2 diabetes. So you have this peripheral insulin resistance that happens in your muscle, your fat, your liver, where your body just doesn’t respond to high levels of blood sugar and it becomes toxic to your body. The brain is a very interesting organ in how it uses energy, how it responds to insulin, and insulin serves a double role.

It’s not only important for managing glucose, but it also seems to be somewhat of a trophic factor that supports synaptic health. And we know neurons need to fire to have proper brain function, and insulin really supports that both from a metabolic standpoint, but then from almost a synaptic plasticity standpoint. And I think that there’s a lot of information coming out that is getting at the core of what insulin resistance in the brain means, what does this mean in the context of Alzheimer’s, and maybe it could be something we could target either pharmacologically or through lifestyle interventions.

IRA FLATOW: The authors are quoted in a press release saying that, quote, “Because the average person is eating insulin spiking foods so frequently, I look at these findings as a problem we’ve created and that we’re making worse.” Unquote. Is it too simple to say that eating differently could solve Alzheimer’s?

SHANNON MACAULEY: No, I don’t think it’s too simple, per se, and I’ll qualify it by saying that I don’t know that changing our diet is going to ward off all the aspects of Alzheimer’s. Alzheimer’s is really the confluence of many different mechanisms over a lifetime. But what I will say is that bad eating habits, poor sleep, inactivity, all comes down to the fact that we’re not taking in energy properly or using energy properly, and this is really targeting these key networks in our brain to wear them out. So I do think that lifestyle modifications can all keep these networks healthier longer, and if nothing else increase our quality of life so we can age in a healthy way.

IRA FLATOW: What I have not heard you say and what I have not heard from reading the research we’re talking about is the word plaques. And I mean, that’s all we hear about Alzheimer’s, is clearing the plaques in the brain. Is that not necessary?

SHANNON MACAULEY: So you can’t have Alzheimer’s disease with our current definition without considering amyloid plaques and neurofibrillary tangles. At autopsy, that’s how we diagnose Alzheimer’s disease. There is a lot of work, including work from my own lab, that has shown that if you have high glucose levels or hyperglycemia, this makes more amyloid beta and pushes the formation of amyloid plaques.

So one way that poor diet can be to poor brain health is through generating too much A beta and forming too much amyloid plaques. But the thing is is that clearing amyloid plaques is insufficient for solving the complexity of Alzheimer’s disease. And again, one of the things that’s coming out with a lot of work nowadays is the what’s the role of glia, what are the astrocytes, the microglia, and the oligodendrocytes doing to contribute to this? And those cell types are very reliant on glucose. They have a lot of different roles, and so it could be that those cells are the ones that really are driving this metabolic phenotype. And it might be, if we can get those cells to respond better, you might not have as many amyloid plaques or neurofibrillary tangles.

IRA FLATOW: You’re of the opinion then that if we follow this metabolism road, we might get some really good results about understanding Alzheimer’s.

SHANNON MACAULEY: Absolutely, and I think you can see that with, again, lifestyle interventions as well as some clinical trials that are repurposing diabetic medications like metformin. Metformin is used to treat type 2 diabetes, but it’s also being looked at as potentially having a helpful role in Alzheimer’s disease. And I think that there is a lot of great medicines out there that people are taking that could help or hinder the development of Alzheimer’s disease. And this is an avenue that we need to investigate further. I think we definitely have realized that removing amyloid plaques is not going to be our cure-all for Alzheimer’s.

And so I really appreciate the NIH’s investment and diverse approaches. There’s a lot where we’re using genetic studies to drive some new avenues into neuroinflammation and what the microglia and astrocytes are doing. So I think that we’re going to see a lot of really innovative approaches to treating Alzheimer’s disease in the next 5 to 10 years.

IRA FLATOW: Dr. Macauley, thank you for this really informative discussion.

SHANNON MACAULEY: My pleasure. Thanks for having me.

IRA FLATOW: Dr. Shannon Macauley, an assistant professor of geriatric medicine. She studies the connection between Alzheimer’s disease and type 2 diabetes at Wake Forrest school of Medicine, Winston-Salem, North Carolina.

Copyright © 2021 Science Friday Initiative. All rights reserved. Science Friday transcripts are produced on a tight deadline by 3Play Media. Fidelity to the original aired/published audio or video file might vary, and text might be updated or amended in the future. For the authoritative record of Science Friday’s programming, please visit the original aired/published recording. For terms of use and more information, visit our policies pages at http://www.sciencefriday.com/about/policies/

Meet the Producer

About Christie Taylor

Christie Taylor was a producer for Science Friday. Her days involved diligent research, too many phone calls for an introvert, and asking scientists if they have any audio of that narwhal heartbeat.

Explore More