Fireside Chat, Interview
Building the Next Generation of Conversational AI
- Strategic Positioning: Sesame is prioritizing the development of a "companion" product over a general-purpose API or utility assistant, viewing natural language (voice) as the next major interface layer for computing, analogous to the shift from command lines to graphical user interfaces.
- Team Scale: The entire software and research team is under 15 people, including core ML and infrastructure engineers, focusing on a "talent-dense" approach rather than scaling headcount.
- Open Source Strategy: Sesame is open-sourcing the Conversational Speech Model (CSM) base model (weights) for research purposes, but explicitly not releasing the full "Maya and Miles" demo, the specific character personalities, or the surrounding system optimizations.
- Technical Limitations (Current Demo): The current Research Preview relies on a transcription step; it does not natively understand audio context (paralinguistics, emotional tone, or silence) and treats speech as a separate generation phase rather than an integrated multimodal understanding.
- Roadmap for Audio Understanding: The near-term research roadmap involves adding an "understanding modality" to pre-trained LLMs to allow native audio input processing, followed by merging this with speech generation into a single transformer that handles both understanding and generation.
- Future Architecture Goal: The long-term objective is a full "duplex" model capable of continuous, frame-level decision-making (approx. 100ms intervals) to handle natural turn-taking, interruptions, and back-channels without heuristic rules.
- Voice Quality vs. Intelligence: The product deliberately trades off some raw reasoning capabilities and "correctness" (e.g., allowing natural speech disfluencies like hesitations or corrections) to achieve a higher degree of human-like naturalness and personality.
- Scalability Insights: Evaluations on the CSM model show that scaling up parameters (1B to 8B) significantly improves long-tail contextual capabilities, such as homograph selection (e.g., "lead" vs. "lead") and consistent accent/voice cloning.
- Hardware Vision: The founders identify smart glasses as the optimal form factor for a "companion interface" due to the need for low-friction, always-available access to computing, mimicking a human companion's presence in the room.
- Differentiation Strategy: Sesame differentiates from larger labs by focusing on "creative taste" and product experience rather than just raw model scale, arguing that the "companion" category requires a level of emotional resonance that general-purpose assistants currently lack.
- Evaluation Methodology: Traditional metrics like Word Error Rate (WER) are deemed saturated; Sesame relies on qualitative "human evaluation" where raters compare model outputs against real human conversation continuations to judge naturalness and personality.
- Future Capabilities: While the current focus is on conversation quality, the long-term product vision includes the companion acting as a mediating layer that can perform multi-step tasks, access external services, and maintain long-term memory of the user relationship.