Connect with us

Artificial Intelligence

‘Audeo’ teaches synthetic intelligence to play the piano


Anybody who’s been to a live performance is aware of that one thing magical occurs between the performers and their devices. It transforms music from being simply “notes on a web page” to a satisfying expertise.

A College of Washington crew questioned if synthetic intelligence might recreate that delight utilizing solely visible cues — a silent, top-down video of somebody taking part in the piano. The researchers used machine studying to create a system, referred to as Audeo, that creates audio from silent piano performances. When the group examined the music Audeo created with music-recognition apps, akin to SoundHound, the apps appropriately recognized the piece Audeo performed about 86% of the time. For comparability, these apps recognized the piece within the audio tracks from the supply movies 93% of the time.

The researchers offered Audeo Dec. 8 on the NeurIPS 2020 convention.

“To create music that sounds prefer it could possibly be performed in a musical efficiency was beforehand believed to be not possible,” mentioned senior writer Eli Shlizerman, an assistant professor in each the utilized arithmetic and {the electrical} and laptop engineering departments. “An algorithm wants to determine the cues, or ‘options,’ within the video frames which can be associated to producing music, and it must ‘think about’ the sound that is taking place in between the video frames. It requires a system that’s each exact and imaginative. The truth that we achieved music that sounded fairly good was a shock.”

Audeo makes use of a sequence of steps to decode what’s taking place within the video after which translate it into music. First, it has to detect which keys are pressed in every video body to create a diagram over time. Then it must translate that diagram into one thing {that a} music synthesizer would really acknowledge as a sound a piano would make. This second step cleans up the information and provides in additional data, akin to how strongly every secret’s pressed and for the way lengthy.

“If we try and synthesize music from step one alone, we’d discover the standard of the music to be unsatisfactory,” Shlizerman mentioned. “The second step is like how a instructor goes over a scholar composer’s music and helps improve it.”

The researchers skilled and examined the system utilizing YouTube movies of the pianist Paul Barton. The coaching consisted of about 172,000 video frames of Barton taking part in music from well-known classical composers, akin to Bach and Mozart. Then they examined Audeo with virtually 19,000 frames of Barton taking part in completely different music from these composers and others, akin to Scott Joplin.

As soon as Audeo has generated a transcript of the music, it is time to give it to a synthesizer that may translate it into sound. Each synthesizer will make the music sound a bit of completely different — that is just like altering the “instrument” setting on an electrical keyboard. For this examine, the researchers used two completely different synthesizers.

“Fluidsynth makes synthesizer piano sounds that we’re accustomed to. These are considerably mechanical-sounding however fairly correct,” Shlizerman mentioned. “We additionally used PerfNet, a brand new AI synthesizer that generates richer and extra expressive music. Nevertheless it additionally generates extra noise.”

Audeo was skilled and examined solely on Paul Barton’s piano movies. Future analysis is required to see how effectively it might transcribe music for any musician or piano, Shlizerman mentioned.

“The aim of this examine was to see if synthetic intelligence might generate music that was performed by a pianist in a video recording — although we weren’t aiming to copy Paul Barton as a result of he’s such a virtuoso,” Shlizerman mentioned. “We hope that our examine allows novel methods to work together with music. For instance, one future software is that Audeo may be prolonged to a digital piano with a digital camera recording only a individual’s fingers. Additionally, by putting a digital camera on prime of an actual piano, Audeo might doubtlessly help in new methods of instructing college students the best way to play.”

Kun Su and Xiulong Liu, each doctoral college students in electrical and laptop engineering, are co-authors on this paper. This analysis was funded by the Washington Analysis Basis Innovation Fund in addition to the utilized arithmetic and electrical and laptop engineering departments.

Story Supply:

Supplies supplied by College of Washington. Authentic written by Sarah McQuate. Word: Content material could also be edited for fashion and size.

Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *