The world of voice AI, with Mati Staniszewski of ElevenLabs

Phonemes function as audio tokens

Voice models predict next phonemes (deconstructed syllable sounds) based on previous audio context and textual input, operating similarly to token prediction in LLMs.

Emergent voice characteristics replace hard-coding

Rather than using preset parameters for accents or emotions like early Bell Labs systems, the model deduces Britishness, enthusiasm, or sadness emergently through neural architecture.

Contextual prediction requires dual processing

The model simultaneously processes text construction and audio waveform generation to understand sentence context and render appropriate emotional inflection and prosody.

Horizontal infrastructure focus

ElevenLabs builds foundational models and infrastructure for voice agents, telephony, and creative tools while avoiding domain-specific vertical applications.

Direct deployment ensures model currency

Working directly with enterprises prevents intermediation risks where customers might remain stuck on outdated model versions instead of accessing weekly capability improvements.

Full-stack voice agent infrastructure

The platform combines TTS, STT, and conversational models with knowledge base integration, telephony connections, and safety monitoring for production deployment.

Consumer deployment lags capability

Despite three years of capable voice technology, integration into cars and phones remains rudimentary due to slow automotive adoption and OS limitations on third-party transcription engines.

ElevenReader fills distribution void

The consumer app enables PDF narration and AI audiobook creation using voices like Sir Michael Caine after traditional distributors blocked AI-generated audio content.

Enterprise adoption precedes automotive

Real-time contextual voice agents are currently deploying in enterprise settings, with in-car offline voice processing expected within 2-3 years.

Specialized audio annotation teams

Built dedicated teams trained specifically to annotate emotional context, accents, and prosody in audio, as generic labelers lacked expertise to describe vocal characteristics.

Internal tools become commercial products

Speech-to-text models initially developed for internal data annotation evolved into commercial offerings, with production feedback loops continuously improving model accuracy.