“If we could create tools and instruments that lead to the emergence of a new music genre, that would be incredible. We want to empower artists to be able to have the tools to do that.”
This is the dream of Neutone’s co-founder, Andrew Fyfe. As a musician himself, he’s taken an unlikely path from touring in a Glasgow band to engineering ‘neural sampling’ in Tokyo. With a PhD background in AI research, he found himself in the right place at the right time just as AI technology burst into the global zeitgeist.
Along with co-founder and CEO Nao Tokui, Neutone produced its first VST plugin called Morpho in 2024—allowing you to morph one sound into another using AI trained models. Since then they have partnered with Roland to embed their AI software into a stompbox interface. Occupying an intriguing and niche corner of the AI music scene, Fyfe hopes that Neutone’s latest invention will work with musicians, not against them.
“We see AI’s role being this extension of what’s possible, it’s something that should be in the loop with humans. It should complement all the processes and music-making practices that already exist. We don’t want to automate any of that away.”
While not yet available to buy, Neutone and Roland have built a prototype AI effects box dubbed Project LYDIA: an anagram of AI and DIY. It’s a nod to Roland’s legacy that includes the AMDEK build-your-own line of effects pedals released in the 1980’s. Roland’s marketing at the time stated “We want to turn passive customers into active musicians”, a sentiment not far from Neutone’s vision for LYDIA. However, almost 50 years later it’s the AI-model that can be custom built, tweaked, or swapped out.
“You can open it up, you can hack it, you can change it”, says Fyfe, “you can change the models, you can change the kind of AI that’s running on it – you can really personalise it and make it your own.”
While it seems that every music company under the sun is adding AI to their software, Neutone are among the few attempting to integrate AI into a piece of hardware. Dedicated to creating real-time performance effects, Neutone have taken Project LYDIA on its own tour of audio developer conferences to get hands-on feedback, including NAAM in the U.S., Superbooth Berlin, and ADC Tokyo.
Through this process, Fyfe realised what a difference the physical form made to people’s understanding of AI technology. “When this technology was brought into this hardware form factor, for a lot of people it just made sense instantly, and to see people’s eyes light up when they saw what was going on, it was so exciting.”
Project LYDIA works by taking an audio signal input and running it through an AI model of another instrument or sound. While the resultant sound you hear is of the AI model—whether it’s drums, voice, or something completely abstract—its shape, dynamics, and nuance is crafted by your performance. Neutone are calling it neural sampling, but another way to think of it is transferring the timbre of one sound to another.
“The simplest description is to imagine taking your voice and making it a drum,” Fyfe explains. “You’re beatboxing with your voice and on the other end you hear djembe percussion or drum kit sounds. It’s a very simplistic way of thinking about it but the possibilities are far beyond just that one-to-one kind of translation.”
Where things get interesting is that Neutone’s AI models can be trained on just about anything, and in the hands of artists that can go to some fascinating places. In one example, the company worked with the artist Pecas to record the sound of her heartbeat and breaths using contact mics inside an anechoic chamber.
“The cool thing is it’s being driven by arbitrary audio inputs”, says Fyfe, “so you could feed a drum sequence and get these pumping heartbeats and breath sounds, essentially transforming the drum loop into these kinds of textures. You can re-texturise the sound into this completely new sound world.”
Neutone has an ongoing artist series where they work with creatives to build unique AI models. Some models are available to buy on their platform with 50% of the profits going direct to the artist they work with. Building an AI model that delivers good results isn’t always easy, and Fyfe describes the process as a bit like ‘alchemy’, adding that “It feels like an artform. You’re crafting these models in a way that’s artistic in itself.”
Luckily the instructions aren’t being kept under lock and key. Neutone have released guides for how to make your own AI model using their service, appealing to musicians who enjoy a higher degree of control, or simply want to experiment with what’s possible.
Having fine-tuned their AI software over several years, Neutone are now focused on finding a way for everything to pack down into a small effects pedal running a Raspberry Pi computer. With significantly less computing power to work with, it’s the kind of engineering challenge that has meant most companies are sticking to digital AI tools that run via cloud computing. Despite the challenge, Fyfe says that real-time performance has always been the goal.
“We always wanted to build workflows that aligned with the live performance aspect and you can only really do that if the AI models are light, on-device and run fast,” says Fyfe, “the plan was for all the products we put out to have AI models run locally on the user’s machine. There’s no talking to a cloud and sending information up there for processing. We’re a big believer in that because of the uncertainty amongst people about how AI is processing their information.”
Scorn for AI has steadily risen over the last few years, but interestingly, surveys continue to show that people are open to working with AI music tools if it’s done ethically. Seeing where things were headed, Neutone initially began working with open source and licensed datasets, and now build datasets in-house. With a team full of musicians and artists, they are clear about where their ethics lie on the subject.
“We believe that artists should be respected. The materials, their IP – everything should be respected. We shouldn’t be taking their material and building AI models on it that may potentially compete with that original source material in some way. Luckily the technologies we’re building don’t really do that because they’re new kinds of instruments.”
In spite of some attempts to remove the musician from the music, Neutone and Roland are ultimately hoping for a different creative relationship. Thankfully, embedding AI into a physical form comes with one major benefit: human input is a guaranteed part of the equation.
The post Neutone and Roland want you to design your own AI model — and play it live appeared first on MusicTech.
