Interview, Fireside Chat
Making Music and Art Through Machine Learning - Doug Eck of Magenta
- Magenta's Core Mission: Google's Magenta project aims to build open-source tools and machine learning models that assist creative people in generating new forms of art and music, rather than replacing them.
- Philosophy of "Failure": The project embraces the Brian Eno concept that the initial "ugly" or broken traits of a new medium (like digital distortion or 8-bit sound) eventually become its signature aesthetic and are emulated as new styles.
- NSynth Sound Generation:
- The team is using deep learning to generate new sounds by exploring a "latent space" or compressed representation of audio that prevents simple memorization.
- Unlike traditional synthesizers, NSynth can interpolate between distinct sounds (e.g., a trombone and a flute) to create novel timbres with unique harmonic properties.
- Musicians find these sounds fascinating because the "glitchiness" arises from the model's confusion, which retains musical essence, rather than from digital clipping.
- SketchRNN and Drawing:
- This model, trained on data from the "Quick, Draw!" game, generates sketches of objects like cats, dogs, and camouflaged items.
- Artists utilize SketchRNN to find "weird" or improbable examples by using the model as a distance measure within its probability space.
- Evaluation Challenges:
- The team currently lacks a robust method to evaluate the quality of generated music and art beyond cherry-picking interesting results.
- A proposed solution involves creating a viral mobile application to gather large-scale human feedback data, similar to collaborative filtering used in music recommendation.
- AI Duet and Improvisation:
- Early models like "AI Duet" function primarily as a call-and-response tool for musicians rather than autonomous melody generators.
- Talented musicians engage with these primitive models by improvising to "follow the leader," treating the AI's limited, often surprising output as a challenge to react to.
- Historical Context of LSTM:
- Doug Eck highlights that while LSTM (Long Short-Term Memory) networks were developed decades ago, their utility was initially limited by hardware constraints.
- Progress was driven by Alex Graves, who persistently refined the model until increased computing power and larger datasets allowed it to succeed in speech and language tasks.
- Limitations of Current Models:
- Vanilla LSTMs struggle to generate long-form hierarchical structures (e.g., chord changes or complex plots) without conditional or hierarchical modifications.
- Current outputs often resemble caricatures of data (e.g., a "Platonic cat") because the models compress variance to fit limited memory.
- Reinforcement Learning (RL) Applications:
- The team is exploring RL to improve generative models by providing scalar rewards for following specific rules (e.g., 18th-century counterpoint) or aesthetic constraints (e.g., "no straight lines").
- This approach allows models to be "tilted" toward specific creative goals without retraining from scratch, combining data-driven realism with user-defined heuristics.
- Future of "Perfect" Pop Music:
- Eck predicts that while AI can likely generate mathematically perfect pop songs, human artists will likely adapt by adding "harder" elements that are difficult to automate, continuing the cycle of innovation.
- The trend suggests that as easy tasks become automated, human creativity will shift toward more complex, unpredictable, or emotionally raw expressions.
- Long-Form Generation Goals:
- A key benchmark for Magenta is generating coherent long-form music and art that handles nested structures, such as chord progressions over minutes rather than seconds.
- This would allow human composers to offload structural decisions to the AI and focus on local details like expressive timing and texture (e.g., the specific "lurching" style of Thelonious Monk).
- Community and Workflow:
- The team emphasizes that the tool loses value if users only "push a button"; creative coding and active participation in the workflow are essential for the cathartic experience of creation.
- Current friction points include the command-line interface, which requires dumping files, and the need for better integration with professional DAWs like Ableton.
- Call to Action:
- The team seeks more "creative coders" who can hack, retrain, and improve the open-source codebase to make the tools more accessible and expressive.
- Users are invited to visit
magenta.tensorflow.orgorg.co/magentato access the code, report issues, and join the discussion list for technical and philosophical feedback.