In this article, we will explore the collaboration between sound designer Simon Pyke, also known as Freefarm, and singer James Buttery, as they worked together on a new album project. The project aimed to create music videos that would visually represent the mood and tone of the songs, while also featuring the lyrics in a unique typographic treatment. To achieve this, the team employed a generative AI system to create the visuals, resulting in videos full of surprises and unexpected outcomes.
Background:
Simon Pyke, known for his work as a sound designer and musical artist, teamed up with singer James Buttery, formerly a member of the electronic music group Darkstar, to collaborate on a new album project. Unlike most of Simon’s previous work, this project took on a more song-based approach, with James providing the lyrics for the tracks. In response to the lyrics, the team wanted to create music videos that would visually represent the mood and tone of the songs, while also incorporating the lyrics in an innovative way.
Design Thinking:
The team’s goal was to build a generative system that would automate the creation of the music video visuals, rather than animating them by hand. They wanted the outcomes to be surprising and unpredictable, avoiding each frame becoming too predictable. The lyrics would appear and animate in response to the music, creating a dynamic and living experience for the viewer. To achieve this, a custom system was created that allowed manual control over the basic structure of how the lyrics appeared and animated, while also incorporating a generative AI layer to bring everything to life.
Additionally, the team wanted to create a system that would allow them to creatively respond to the themes and feel of the music. In this particular instance, the song focused on a cycle commute through the urban environment of London streets, and the team found inspiration in brutalist architecture as a visual cue.
Challenges:
One of the main challenges the team faced was striking a balance between legibility and abstraction in the visuals. They wanted the lyrics to exist in a world that reflected the music, rather than appearing too literal or clinical. However, pushing them too far into abstraction could result in the words melting away and becoming meaningless. The team experimented with various approaches and configurations before finding a balance that they were happy with. Interestingly, when viewed independently, many frames of the animation were not particularly legible. However, when viewed as a moving image sequence, the words came to life and became discernible.
Favorite details:
The team highlights the fact that generative AI imagery is often heavily curated, with only the “good” results being shared and the rest discarded. However, in this project, they wanted to create thousands of images per video, so they had to ensure that the system could produce variety while minimizing undesirable images. A significant amount of work went into tuning the typographic approach and the generative AI system’s parameters to create videos full of surprises, even for their creators. They wanted the videos to maintain a sense of cohesion, legibility, and the desired aesthetic throughout.
New lessons:
Throughout the project, the team explored various approaches to the generative AI element and learned a great deal about working with this emerging technology alongside traditional animation techniques. They discovered how to retain specific structures and patterns in the generated AI text-to-image process, which is less commonly seen in examples of this technology.
Time constraints:
As these were music videos for self-released tracks, the team had to be mindful of the time spent creating them. Generative systems are enjoyable to work with and can easily consume a lot of time, so the team set constraints to prevent them from getting lost in the process of creating endless variations. They also decided to follow a simple animation formula that could be created procedurally, rather than animating everything by hand. This decision resulted in each video having its own rhythm, which may not have emerged if they had been hand-animated.
Conclusion:
The collaboration between Simon Pyke and James Buttery resulted in a unique album project that combined music, visuals, and generative AI technology. By employing a custom system, the team was able to create music videos that visually represented the mood and tone of the songs, while also incorporating the lyrics in a visually engaging typographic treatment. The use of generative AI added an element of surprise and unpredictability to the visuals, resulting in videos that both surprised and delighted their creators. Overall, this project demonstrates the exciting possibilities of merging traditional music and art forms with emerging technologies like generative AI.