What happens when AI enters the concert hall

microphone on stage in concert hall theater
Concert hall. (Photo: Envato Elements)
When composer and vocalist Jen Wang took the stage at the Monk Space in Los Angeles to perform Alvin Lucier’s “The Duke of York” (1971) earlier this year, she sang with a digital rendition of her voice, synthesized by artificial intelligence.اضافة اعلان

It was the first time she had done that. “I thought it was going to be really disorienting,” Wang said in an interview, “but it felt like I was collaborating with this instrument that was me and was not me.”

Isaac Io Schankler, a composer and music professor at Cal Poly Pomona, conceived the performance and joined Wang onstage to monitor and manipulate Realtime Audio Variational autoEncoder, or RAVE, the neural audio synthesis algorithm that modeled Wang’s voice.

RAVE is an example of machine learning, a specific category of artificial intelligence technology that musicians have experimented with since the 1990s — but that now is defined by rapid development, the arrival of publicly available, AI-powered music tools and the dominating influence of high-profile initiatives by large tech companies.

Schankler ultimately used RAVE in that performance of “The Duke Of York,” though, because its ability to augment an individual performer’s sound, they said, “seemed thematically resonant with the piece.” For it to work, the duo needed to train it on a personalized corpus of recordings. “I sang and spoke for three hours straight,” Wang recalled. “I sang every song I could think of.”

Antoine Caillon developed RAVE in 2021, during his graduate studies at IRCAM, an institute founded by composer Pierre Boulez in Paris. “RAVE’s goal is to reconstruct its input,” he said. “The model compresses the audio signal it receives and tries to extract the sound’s salient features in order to resynthesize it properly.”

Machine learning then, and now
Tina Tallon, a composer and professor of AI and the arts at the University of Florida, said that musicians have used various AI-related technologies since the mid-20th century.

“There are rule-based systems, which is what artificial intelligence used to be in the ’60s, ’70s, and ’80s,” she said, “and then there is machine learning, which became more popular and more practical in the ’90s and involves ingesting large amounts of data to infer how a system functions.”

Today, developments in AI that were once contained to specialized applications impinge on virtually every corner of life and already impact the way people make music. Caillon, in addition to developing RAVE, has contributed to the Google-led projects SingSong, which generates accompaniments for recorded vocal melodies, and MusicLM, another text-to-music generator.

Innovations in other areas are driving new music technologies, too: WavTool, a recently released, AI-powered music production platform, fully integrates OpenAI’s GPT-4 to enable users to create music via text prompts.

For Tallon, the difference in scale between individual composers’ customized use of AI and these new, broad-reaching technologies represents a cause for concern.

“We are looking at different types of data sets that are compiled for different reasons,” she said. “Tools like MusicLM are trained on data sets that are compiled by pulling from thousands of hours of labeled audio from YouTube and other places on the internet.

“When I design a tool for my own personal use, I’m looking at data related to my sonic priorities. But public-facing technologies use data sets that focus on, for instance, aesthetic ideals that align more closely with Western classical systems of organizing pitches and rhythms.”

Bias in music-related AI
Concerns over bias in music-related AI tools do not stop at aesthetics. Enongo Lumumba-Kasongo, a music professor at Brown University, also worries about how these technologies can reproduce social hierarchies.

“There is a very specific racial discourse that I’m very concerned about,” she said. “I don’t think it’s a coincidence that hip-hop artistry is forming the testing ground for understanding how AI affects artists and their artistry given the centuries-long story of co-optation and theft of Black expressive forms by those in power.”

The popularity of recent AI-generated songs that mimicked artists like Drake, the Weeknd, Travis Scott and others have animated Lumumba-Kasongo’s fears. “What I’m most concerned about with AI Drake and AI Travis Scott is that their music is highly listenable,” she said, “and calls into question any need for an artist once they’ve articulated a distinct ‘voice.’”

For Schankler, there are key differences between using RAVE to synthesize new versions of a collaborator’s voice and using AI to anonymously imitate a living musician. “I don’t find it super interesting to copy someone’s voice exactly, because that person already exists,” they said. “I’m more interested in the new sonic possibilities of this technology. And what I like about RAVE is that I can work with a small data set that is created by one person who gives their permission and participates in the process.”

Composer Robert Laidlow also uses AI in his work to contemplate the technology’s fraught implications. “Silicon,” which premiered in October with the BBC Philharmonic under Vimbayi Kaziboni, employs tools to explore themes drawn from the technology’s transformative and disruptive potential.

Laidlow described “Silicon” as “about technology as much as it uses technology,” adding: “The overriding aesthetic of each movement of this piece are the questions, ‘What does it mean for an orchestra to use this technology?’ and ‘What would be the point of an orchestra if we had a technology that can emulate it in every way?’”

The work’s entirely acoustic first movement features a mixture of Laidlow’s original music and ideas he adapted from the output, he said, of a “symbolic, generative AI that was trained on notated material from composers all throughout history.” The second movement features an AI-powered digital instrument, performed by the orchestra’s pianist that “sometimes mimics the orchestra and sometimes makes uncanny, weird sounds.”

In the last movement, the orchestra is accompanied by sounds generated by a neural synthesis program called PRiSM-SampleRNN, which is akin to RAVE and was trained on a large archive of BBC Philharmonic radio broadcasts. Laidlow describes the resulting audio as “featuring synthesized orchestral music, voices of phantom presenters and the sounds the artificial intelligence has learned from audiences.”

The size of “Silicon” contrasts with the intimacy of Schankler and Wang’s performance of “The Duke of York.” But both instances illustrate AI’s potential to expand musical practices and human expression. And, importantly, by employing small, curated data sets tailored to individual collaborators, these projects attempt to obviate ethical concerns many have identified in larger-scale technologies.

Read more Technology
Jordan News