AI Models & Platforms
Baseten Opens Voice Agent Stack and Realtime Voice in Private Preview

Key PointsCo-locating orchestration with speech models removes network round trips that remote…
View more
- Co-locating orchestration with speech models removes network round trips that remote orchestrators add per turn.
- Baseten's stack lets operators scale each pipeline stage independently instead of duplicating the full pipeline.
- OpenAI SDK clients cannot connect directly because Baseten's endpoint does not serve the appended realtime path.
Baseten announced the Voice Agent Stack and Baseten Realtime Voice on November 16, 2026, opening both to private preview through a waitlist and reporting roughly one-second median turn latency across up to 56 concurrent sessions in its test setup. The stack co-locates speech models and orchestration, lets customers choose models independently and scales each pipeline component separately.
Co-Located Infrastructure
The Voice Agent Stack runs on Baseten Dedicated Inference, placing the orchestrator and models in one cloud region over an in-cluster network. Baseten said a remote orchestrator can add hundreds of milliseconds of network round trips per turn even when the speech-to-text, language and text-to-speech components are all hosted on its platform. The stack also supplies inference-runtime optimization, inter-model networking, traffic autoscaling, cross-cloud failover and model post-training support.
Realtime Voice is the reference implementation, running a cascade of Baseten-optimized open-weight models for speech-to-text, language processing and text-to-speech with a custom Pipecat-based orchestrator. Its optional delegation design keeps a fast, co-located talker model handling conversation while a stronger external thinker model asynchronously manages reasoning, web search, decisions and tool calls. Server-side tools execute on the co-located orchestrator, which can also call remote Model Context Protocol servers, avoiding extra client round trips; client-side tools are supported as well.
Model Control and API Compatibility
Customers can choose the speech-to-text, language and text-to-speech models independently, combine dedicated components with third-party APIs and move individual pieces onto dedicated infrastructure over time. Post-training options include adapting transcription to industry jargon, tuning language-model behavior for calls and supervised fine-tuning of speech with a customer’s voice data. A session can switch its language model, voice or transcription route without redeployment.
The API matches the core OpenAI Realtime event protocol and supports audio-in/audio-out and text-in/audio-out modes, alongside transcription keyword boosting, configurable acknowledgments and voice cloning. Compatibility has practical limits: clients must accept a full WebSocket URL. OpenAI’s Python SDK cannot connect directly because it appends /realtime to its base URL, a path Baseten’s endpoint does not serve. Browser clients need a backend to hold the API key and relay the socket because browsers cannot set the required Authorization header.
Latency and Cost Under Concurrency
Baseten reported median time to first audio rising from 0.94 seconds with one concurrent call to 1.14 seconds at 56. The latency tests used contexts up to 16,000 tokens; longer contexts can raise turn latency, while configurable summarization can hold each session to a target context length. The test configuration used Qwen3-ASR on one RTX-6000, Gemma-4-31B on one B200, Qwen3-TTS on another RTX-6000 and GLM-5.3 Fast through Baseten Model APIs for delegation.
Each component scales independently against its own concurrency target, allowing operators to add capacity to the stage that saturates instead of duplicating the full pipeline. The design is intended to preserve strict p90 and p95 latency tails while keeping GPU utilization high. Baseten estimated $0.012 per audio minute at 32 concurrent sessions, excluding the per-token cost of reasoning-model delegation, and $0.008 per call-minute at 56 sessions including Model API delegation, under its stated utilization assumptions. Dedicated pricing amortizes hardware across additional sessions, making sustained utilization central to those economics.
An independent speech-to-text benchmark from Coval provides component-level context. Its 24-hour leaderboard, with data timestamped 17:30 UTC on November 16, 2026, ranked Baseten-hosted Qwen3 ASR 1.7b second by median Time to Final Segment at 33 milliseconds, with a 2.1% word error rate over 456 samples. The metric starts at a shared end-of-speech point and ends when the final transcript arrives, excluding the time a live system spends detecting that the turn has ended. It therefore does not measure the full agent response.
Baseten said future work includes more complex orchestration flows, easier migration of existing voice pipelines, native speech-to-speech full-duplex systems, omni models with dedicated speech-generation heads and new scheduling primitives across the cascade. The launch makes orchestration the control point for voice-agent economics: model choice, tool execution and component-level capacity are handled together, giving operators several ways to tune latency and GPU use without rebuilding the full pipeline.












