The world of voice AI, with Mati Staniszewski of ElevenLabs
TL;DR
ElevenLabs CEO Mati Staniszewski explains how modern voice AI models predict phonemes with contextual awareness rather than using hard-coded parameters, enabling emergent properties like accents and emotions, while discussing the company's platform strategy and the deployment gap between capable models and consumer applications.
🧠 Technical Architecture 3 insights
Phonemes function as audio tokens
Voice models predict next phonemes (deconstructed syllable sounds) based on previous audio context and textual input, operating similarly to token prediction in LLMs.
Emergent voice characteristics replace hard-coding
Rather than using preset parameters for accents or emotions like early Bell Labs systems, the model deduces Britishness, enthusiasm, or sadness emergently through neural architecture.
Contextual prediction requires dual processing
The model simultaneously processes text construction and audio waveform generation to understand sentence context and render appropriate emotional inflection and prosody.
🏗️ Platform Strategy 3 insights
Horizontal infrastructure focus
ElevenLabs builds foundational models and infrastructure for voice agents, telephony, and creative tools while avoiding domain-specific vertical applications.
Direct deployment ensures model currency
Working directly with enterprises prevents intermediation risks where customers might remain stuck on outdated model versions instead of accessing weekly capability improvements.
Full-stack voice agent infrastructure
The platform combines TTS, STT, and conversational models with knowledge base integration, telephony connections, and safety monitoring for production deployment.
📱 Deployment Gap 3 insights
Consumer deployment lags capability
Despite three years of capable voice technology, integration into cars and phones remains rudimentary due to slow automotive adoption and OS limitations on third-party transcription engines.
ElevenReader fills distribution void
The consumer app enables PDF narration and AI audiobook creation using voices like Sir Michael Caine after traditional distributors blocked AI-generated audio content.
Enterprise adoption precedes automotive
Real-time contextual voice agents are currently deploying in enterprise settings, with in-car offline voice processing expected within 2-3 years.
📊 Data Innovation 2 insights
Specialized audio annotation teams
Built dedicated teams trained specifically to annotate emotional context, accents, and prosody in audio, as generic labelers lacked expertise to describe vocal characteristics.
Internal tools become commercial products
Speech-to-text models initially developed for internal data annotation evolved into commercial offerings, with production feedback loops continuously improving model accuracy.
Bottom Line
Voice AI's immediate opportunity lies not in improving core models but in closing the deployment gap by integrating existing capable technology into consumer devices and enterprise workflows through full-stack platforms.
More from Stripe
View all
Tech analyst Philip Klöckner in conversation with Conor McNamara
Tech analyst Philip Klöckner argues that AI is driving a paradigm shift in software development and business formation, creating a bifurcated economy where AI-native startups operate with radically leaner teams and no legacy software dependencies, while incumbent enterprises face gradual disruption buffered by demographic shifts and organizational inertia.
A conversation with Alan cofounder and CTO Charles Gorintin
Charles Gorintin, CTO of Alan, recounts the company's decade-long journey from Silicon Valley roots to becoming a European healthtech leader with 4 million members, detailing their strategy of aggressive early internationalization, AI transformation through the medical agent MO, and the strategic imperative of building European technological sovereignty via Mistral.
Barney Hussey-Yeo in conversation with John Collison
Cleo founder Barney Hussey-Yeo discusses building an AI financial assistant since 2016, leveraging humor and proactive agentic technology to optimize financial decisions for the 99% of consumers living paycheck to paycheck, while arguing that vertical AI agents will outperform general LLMs in specialized domains like personal finance.
10 Years of Stripe France: The tech renaissance and what’s next
French tech leaders reflect on the ecosystem's transformation from early 2000s corporate culture to today's AI-driven renaissance, highlighting how reduced capital barriers and improved infrastructure are reshaping entrepreneurship.