Spectral Resonator and Spectral Time are two devices introduced in Ableton Live 11. Both break sound up into partials and process it in ways that range from melodic, lush, spooky and futuristic, to stuttery, dissonant, glitched-out and metallic. Though the resulting sounds from each device are distinct, the two are connected by their common ground of being based on the fast Fourier transform (FFT).
The Fourier transform has been around a long time, but this is the first time Ableton is including this technology in Live. Natalye Childress spoke with the product owner and some developers past and present to give you an inside look at the history and inspiration behind these devices, as well as share tips and tricks for how to best use them in your own music-making endeavors.
Breaking down the signal
The Fourier transform is a mathematical algorithm with applications across engineering, image processing, optics and more. Within a musical context, it’s used in signal processing to break signals down into their various parts, or frequencies. It’s named after French mathematician Joseph Fourier who, in 1822, laid the groundwork for what later became known as the Fourier series.
Live 11’s new Spectral devices are based on the fast Fourier transform (FFT), which was popularized for use in audio in the nineties as digital processing became more powerful. FFT is often used interchangeably in name with the discrete Fourier transform (DFT), because they’re essentially the same thing. The difference is that FFT uses calculations quick enough to be used in real time.
“The problem with the Fourier transform and related technology is that, for a long time, it was an offline process,” says Product Owner Christian Kleine. “You couldn’t use it in real time. Instead, you had to send a signal in, process it, wait, and render it to disk or similar. Real-time processing only became possible because of increased computing power.”
Live 11’s Spectral Resonator audio effect on vocals
When a Spectral device is used in Live 11, it performs an FFT on the audio it’s given, transforms it into spectral information, tweaks that information in the form of effects, and converts it back to audio with an inverse fast Fourier transform (IFFT).
“Sound is made up of sine waves, each of which has a certain amplitude, frequency and phase,” says Christian. “These three components allow you to basically create any sound. So you can take a sound and dissect it into single sine waves, but the other way around, you can put them back together, and in the process, dramatically influence the shape of the sound.”
For an in-depth look at the mathematics behind this technology, especially if you’re more of a visual learner, this website by Jack Schaedler – who is on the Ableton Learning Team – is a good place to start.
From the lab to your home
“When it came out, it was mainly an academic discovery, as many of these things are,” says Christian, pointing to how using the Fourier transform was initially limited to people with access to university labs. However, once home computers became more affordable and commonplace, it was something people could do with fewer barriers.
Even so, this kind of technology stayed rooted in the academic world, with one exception: It appealed to many avant garde and modernist musicians. For example, INA GRM, a France-based audiovisual institute and music research group, is associated with modern avant garde composers, many of whom began using the Fourier transform in their music.
For Ableton, there was no external pressure or specific drive to make the Spectral devices, beyond wanting to add a new dimension of sound to Live. Christian likens the devices to spices or ingredients that create a new flavor profile.
“If Ableton is a kitchen, and the devices are different ways of cooking up things, with the Spectral devices, you now have a completely new taste,” he says.
He goes on to clarify: “The technology’s not so new itself, but it’s a new type of device – for us – which does different things. Everything about this technology is difficult, but the results can be super interesting. And that’s one of the main reasons why we decided to make these devices: Because they let you easily achieve things you can’t otherwise achieve at all.”
Robert Tubb, a developer who helped create these devices, agrees. “A lot of these spectral devices are used in academic contexts or the experimental world, and everything out there seems to be detailed and hard to use. What we wanted to do is be inspired by these experimental devices but make them more accessible and a bit more immediate. Sometimes you get these huge suites of spectral plugins and you have to spend days trying to find the good stuff among all the blippy-bloppy sort of stuff. And we set out to locate the sweet spots in these things.”
Spectral effects in popular music
One of the most well-known examples of a song that relies on FFT processing is Aphex Twin’s “ΔMi−1 = −αΣn=1NDi[n] [Σj∈C[i]Fji[n − 1] +Fexti[[n−1]].” After its release, some curious listeners ran the track through a spectrogram to discover an apparition of the artist’s face at the end.
Another example is Venetian Snares’ “Look”, which includes spectrally encoded pictures of cats.
While these effects have been used in popular music, the instances above vary a bit from the new devices. More specifically, as former Ableton Sound Team developer Ian Hobson says, “They’re taking pictures, converting the picture data directly into spectral information, and then using IFFTs to convert them to audio – they visually show that strange and interesting things can happen in the spectral domain.”
There are many works that use spectral effects without relying on images in the academic and avant garde realms, as Christian mentioned earlier.
One such example is “Wind Chimes” by Denis Smalley, an electroacoustic composer from New Zealand who describes his musical approach and style as spectromorphological. For a deep dive into musical structure in the spectral space, his chapter on “Spectro-morphology and Structuring Processes” can be found in the book The Language of Electroacoustic Music.
Another song making use of these techniques is “Puzzle Wood” by British composer, sound artist, and researcher Natasha Barrett.
However, unless you know what you’re listening for, pinpointing specific spectral sounds in a song is not particularly easy. “They're a bit like special effects in movies,” says Hobson, “in that they're often most effective when you don't notice them.”
And outside of the use cases above, “the technology is widely used for sound restoration: for declicking or removing vocals,” adds Christian. “It’s quite often used to make sound polished or to remove certain artifacts from the sound, but not so much for creative misuse – which is our intention.”
Making the impossible possible
So this technology can be applied in ways that are playful, practical, or make the impossible possible. But what do the new devices actually do?
Whereas Spectral Resonator morphs signals and adds color and texture to sounds, Spectral Time freezes and delays the audio.
Spectral Resonator – not to be confused with the preexisting Resonator effect in Live – is particularly good at adding harmonics to a sound or taking them away. One of its highlights is the MIDI sidechain feature, which allows percussive parts to be played as melodic instruments.
“It’s fairly novel,” says Robert, likening it to a more flexible version of a vocoder, which you can read up on if you’re interested in the history of spectral audio signal processing. “There are things that do similar stuff — like plug-ins that use modal synthesis, which is having loads of resonators that you can set the frequency of – but I don’t think anyone’s done that in the spectral domain.”
The effect applied polyphonically sounds full and almost metallic
The effect applied to pads gives the sound a glimmery, shiny feel
With Spectral Time, the name is a nod to the fact that all of the effects it can achieve are somehow related to freezing or delaying sound. Christian likens it to the experience of watching a movie. “When you stop the movie, it freezes the frame. With music, you stop it and the sound is gone. But with this technology, you can stop the sound but it keeps going in this ‘frozen’ state.”
For example, when used with a foot pedal, you can sustain melodic parts while playing live. The freeze and delay properties can be used separately or in combination, as in the sound clips below.
A straightforward guitar melody
The addition of delay gives the melody a glitchy, shimmery, washed-out effect
The sound is sustained when the freeze effect is used
Delay and freeze combine to create a multiphonic, resonating quality of sound
Spectral Sonogram
Accompanying the devices themselves is the sonogram, which shows the algorithm in its visual form.
“If you know how to read it, you can learn more about the effects just by looking at the dry and the wet signals and what’s happening to them,” Christian says, noting that they’re different colors. “If you send in a sine wave or white noise and you tweak the effects, you can see what’s there and how the signal is treated. And because of this spectral treatment, which is complicated, you can also see this complicated treatment on the sonogram.”
Sending a sine wave through Spectral Resonator for a closer look at the sonogram visualizations
But at the same time, he cautions against relying on it too much. While it can be educational, it can be folded away if it’s too distracting.
Practical applications
When asked the best way to use these devices, Christian pauses. “Nowadays, everything’s super prescriptive, but this isn’t exactly the case here,” he says. “You can do a lot of stuff with these devices, but you need to find your own uses.”
That said, he does have a few recommendations.
“They’re amazing on voices and how a voice can change in terms of character. If you want to surprise your friends with an alien voice, use these devices,” he says. Because the human voice is so complex, it’s interesting to see how the devices handle the complexities and what the end result is.
“With Spectral Resonator, you can do vocoder type effects, or just have it quite subtly in the background,” says Robert. “For example, you can use mostly your own vocals, but have the resonance further back to provide a kind of drone underneath things.”
Another instrument Christian recommends is a classic piano. Because our ears are already familiar with how voices and pianos sound, the contrast is much starker, making it easier to hear the difference between an original sound and one treated with Spectral devices.
“The problem with electronic music is that there’s often no context,” he says. “So you have a sound which is weird, and then you put it into something else which is also weird. And then you have a weird sound made out of a weird sound.”
But with common sounds, you’re starting with something familiar and hearing clearly how it can be completely different.
Robert also suggests playing around with the Spectral Resonator on guitar to create something with a drone-y, ambient, slow-evolving edge. “With a clean electric guitar line and lots of notes set up in the Spectral Resonator, it’s a bit like having a sitar: When you hit a single note, all the other notes are excited and droning away in the background.”
Additionally, he notes you can set the sensitivity of the notes, which makes it so soft notes don’t trigger the freeze in Spectral Time, while hard notes do.
You can even duplicate the devices, playing multiple versions of each one or using both effects together, but he cautions against overdoing it.
“You can do something nice using two Spectral Resonators, where you have one set of frequencies in the first Resonator and a different set of frequencies in the second one, and the first one will only excite the second one where they match up – because you have to have a frequency present in the input to make stuff happen in the output,” Robert explains. “But it gets pretty abstract quite quickly if you start stacking them up.”
Proudly digital
When describing the essence of the devices, Christian says they can be summed up in one word: digital. “Over the past few years,” he says, “there has been this push toward analog, analog, analog — everything needs to be analog. Which is great, because I love analog. Everybody does. It’s warm, it’s fuzzy, it’s kind of quirky. Analog is cool. But digital is cool too. And these devices are really, really digital. They’re as digital as it can get.”
He notes that the Spectral devices are a welcome new addition for users looking to elevate their music with new tools and contexts.
“We’re making devices that enrich the way people can use Ableton Live and make sound. If you want to find your own language in music, you need to go to places where you don’t know what to expect,” Christian says. “And these devices do this.”