Magenta is a research group within Google, exploring how machine learning can help musicians in new ways. One of their first projects was NSynth a machine learning algorithm that uses deep neural networks to learn the characteristics of sounds, then create thousands of completely new sounds based on these characteristics. To make the algorithm more easily accessible, we took the sounds that NSynth generates and put them into a musical instrument that enables musicians to play them in a more intuitive way. We call it NSynth Super.

We had to find a way to organize the over 100,000 output sounds of the NSynth algorithm and put them into an instrument to make it intuitive for musicians to play.

Using dials in each corner of NSynth Super, artists can select up to four source sounds (out of sixteen options, from a clavichord to a car engine) to define the space of sounds they’d like to explore. By dragging their fingers across the space between these points, they can play thousands of unique sounds which combine all four of their acoustic qualities.

It fits seamlessly with a musicians’ existing equipment. All they need to do is plug in a MIDI controller, like a keyboard, and use the touchscreen to explore new sounds.

We wanted lots of musicians to access NSynth Super, so we made all of the source code, schematics, and design templates available for download on GitHub.

We explored how we can use machine learning techniques (like dimensionality reduction) to create a new interface for the multidimensional space of sounds generated by the algorithm.

The easiest way for musicians to create is through dedicated hardware interfaces, and this is what we did for the output of the NSynth algorithm. We created over 100k new sounds with the algorithm and put them in a format that musicians intuitively know how to use, and can quickly be expressive with.

The interface also unlocks new possibilities for performance, allowing people to easily move between worlds of sound by swiping the screen, rather than manually adjusting multiple parameters.

Without the interface we created, this would be a big folder of sound files. Instead it’s something that can be used to make music.