Streaming TTS for real-time voice, how it actually works

"Streaming TTS" is the marketing label every text-to-speech vendor uses now, and it covers a wide range of architectures, some of which actually stream and some of which batch internally and stream the response chunks. This post is a tour through what real streaming TTS looks like, why the architectural choices behind sub-150 ms TTFB are non-obvious, and how to evaluate whether a vendor is selling you a streaming model or a batch model with chunked HTTP responses.

The post is for engineers building voice applications: voice agents, IVR systems, real-time accessibility tooling, in-app voice copilots. If you're shipping a TTS into a non-real-time context (audiobooks, video narration, batch dubbing) the architecture below still applies but the latency targets are looser; you can skip to the "When batch is fine" section near the end.

What "streaming" should mean

A streaming TTS is one where the model generates audio output incrementally as text input arrives, and the audio bytes are sent to the client as they become available. The two important properties:

• Time to first byte (TTFB): the time between sending the request and receiving the first audible chunk of audio. This is what determines how natural a voice agent feels.
• Sustained throughput: the rate at which subsequent chunks arrive after the first one. If the model takes 100 ms to produce the first chunk and then pauses for 800 ms before the next one arrives, the user hears a stutter even though TTFB looks good.

A vendor that advertises "streaming" without disclosing both numbers is reporting whichever number flatters them. Real streaming TTS hits both: low TTFB and steady chunk-to-chunk latency.

The naive architecture: full-utterance batch

The simplest TTS pipeline takes a full sentence as input, runs the entire model end-to-end, and returns the full WAV/MP3 file. It works, it's easy to deploy, and it's wrong for any real-time use case.

Why? The latency floor is set by the time the model takes to generate the entire utterance, plus the time to encode and transmit the resulting audio file. For a 5-second utterance, the model has to render all 5 seconds of audio before sending the first byte. Even on a fast model, this is 800-1500 ms of TTFB. Voice agents fail at this latency; the caller experiences the model as "the agent is thinking" rather than "the agent is talking".

Most TTS vendors started here because batch is what the academic literature builds. The transition to streaming required architectural changes that didn't exist in the published research until 2023-2024.

Stage 1: chunked HTTP response (looks like streaming, isn't)

The first cut at "streaming" most vendors shipped was: run the model in batch mode, but split the resulting audio file into 100 ms chunks and send them via HTTP chunked transfer encoding. The chunks arrive one at a time so the client thinks it's streaming, but the model still rendered the full utterance before the first chunk shipped.

How to spot this: TTFB on a 1-second utterance and TTFB on a 5-second utterance are roughly the same (because both are dominated by the model's full-utterance render time). Real streaming has TTFB that's largely independent of utterance length.

This architecture is fine for non-real-time use cases (audiobook chapter playback, podcast generation) where the chunked transfer just lets the user start listening before the full file finishes downloading. It's not fine for voice agents.

Stage 2: per-phoneme streaming (the real version)

A genuine streaming TTS generates audio incrementally as it processes the input text. The architectural choices that make this work:

Autoregressive decoding with speculative input

The model emits audio frames one at a time, conditioned on the text it's already seen plus the audio it's already generated. As soon as it has generated the first 50-80 ms of audio (typically 4-6 audio frames at 50 Hz frame rate), it can ship that to the client while the next frames are still being decoded.

The speculative input piece is the trick: the model assumes the rest of the text input will arrive (or has already arrived in the WebSocket buffer) and starts generating before the full input is parsed. If the input changes mid-stream (the user types more text into a voice-cloning interface, or an LLM generates more tokens), the model conditions on the new input from that point forward.

Streaming vocoder

The neural vocoder (the component that turns the model's mel-spectrogram or token sequence into actual audio samples) has to support streaming inference. Most of the published vocoders (HiFi-GAN, BigVGAN, WaveNet) can run in streaming mode but require architectural tweaks: causal convolutions only (no future context), bounded receptive field, and warmup handling for the first few frames where context is incomplete.

Hakim Fast v1.3 uses a streaming variant of HiFi-GAN with a 5-frame warmup; the first audio chunk includes a 100 ms blend region where the warmup frames are crossfaded into the steady-state output. This is invisible at the audio level but it's what keeps the TTFB number honest while preserving voice quality.

Frame-level inference scheduling

The model has to schedule frame generation across GPU cores so the generation rate exceeds the playback rate. If the model generates audio at 0.8x real-time, the buffer drains and the user hears stutters. If it generates at 1.5x real-time, the buffer fills with too much pre-rendered audio and the cancel-on-interrupt latency increases. The sweet spot is 1.1-1.2x real-time: enough headroom that occasional GPU contention doesn't drain the buffer, but not so much that interrupt cancellation gets sluggish.

Transport: WebSocket vs. HTTP/2 vs. gRPC

Streaming TTS clients connect over one of three transports:

• WebSocket: bidirectional, symmetric framing, browser-native. The dominant choice for voice agents because the agent platform also uses WebSocket for caller audio. A single connection can carry the input text in one direction and the output audio in the other.
• HTTP/2 chunked response: simpler infrastructure, no separate WebSocket fleet. The client sends a single POST request and the server streams the response over the same connection. Works for unidirectional streaming (text in once, audio out as a stream) but doesn't handle mid-stream input updates.
• gRPC streaming: typed, bidirectional, designed for backend-to-backend streaming. Less common for browser clients but useful for agent-platform-to-TTS connections.

Hakim ships both WebSocket and HTTP/2 chunked; WebSocket is the default for voice agents, HTTP/2 chunked for batch-adjacent use cases (real-time narration of an LLM-generated story, for example) where mid-stream input updates aren't needed.

Mid-stream cancel: the voice agent killer feature

A streaming TTS that can't be cancelled mid-stream is unusable for voice agents. When the caller starts speaking again, the agent has to stop the TTS within ~50 ms or the agent's old response keeps playing while the user is mid-sentence. That's the failure mode where users say "this AI is dumb"; in reality it's the TTS layer that's the problem.

Cancel semantics that work in production:

• The client sends a cancel frame on the WebSocket. The TTS stops generating new audio frames within 30-50 ms.
• The in-flight audio chunk (the one currently being encoded and transmitted) flushes to the client; the client decides whether to play it or drop it.
• The connection becomes available for the next request without reconnecting; the existing voice cache, vocoder warmup state, and audio context stay warm.

The reconnection-free path is what keeps the next-request TTFB low. If every cancel forces a reconnect, the next request has to re-do the WebSocket handshake (50-100 ms round-trip plus TLS) before TTS can start; effective TTFB on the new request degrades by 100-200 ms. Hakim's WebSocket endpoint maintains the connection across cancel-resume cycles for exactly this reason.

Buffer management on the client

Even with a server that streams perfectly, a client that buffers wrong will produce stutters. The two failure modes:

• Underrun: the client plays audio faster than chunks arrive. Stutters become audible as gaps.
• Overrun: the client buffers too many chunks before starting playback. TTFB-perceived-by-user is worse than TTFB-measured-at-network because the client added 200-500 ms of buffering.

The right buffer size depends on the network. For a wired or 5G connection with stable latency, 2-3 chunks (100-150 ms) is enough. For a 3G or congested 4G connection where chunk-to-chunk jitter is 50-150 ms, 4-6 chunks (200-300 ms) is more robust. The client should adapt the buffer size based on observed jitter, not pin it.

The browser-native Web Audio API has a AudioBufferSourceNode for chunked playback, but it doesn't handle adaptive buffering. Production voice agents typically use the Web Audio API's AudioWorkletNode with a custom buffer manager that tracks jitter and adapts.

How to evaluate whether a TTS is genuinely streaming

Five tests that distinguish streaming from chunked-batch:

1. TTFB vs. utterance length: send a 1-second utterance and a 5-second utterance, measure TTFB. Streaming TTS has TTFB roughly independent of utterance length (within 20-50 ms). Chunked-batch has TTFB that scales with utterance length.
2. Time to last byte vs. TTFB: measure the time between TTFB and the last audio byte. For a 5-second utterance, streaming TTS delivers the full audio in ~5 seconds (real-time), with a 500-1000 ms tail after the visible end. Chunked-batch delivers the full audio in 1-2 seconds (post-render burst).
3. Mid-stream cancel latency: send a 5-second utterance, send cancel after 1 second, measure how quickly the audio stream stops. Streaming TTS stops within 30-100 ms. Chunked-batch keeps sending pre-rendered chunks for 500-2000 ms.
4. Mid-stream input update: open a WebSocket, send text in two chunks 200 ms apart, measure whether the audio reflects the second chunk's content. Streaming TTS picks up the new text. Chunked-batch ignores it (the model already started rendering).
5. Connection persistence across requests: close a request, immediately send another, measure new-request TTFB. Streaming TTS keeps the connection warm; new-request TTFB is similar to first-request TTFB. Chunked-batch on plain HTTP forces a reconnect; new-request TTFB is 100-200 ms worse.

If a vendor's TTS fails three or more of these tests, the "streaming" label is marketing.

When batch is fine

Streaming architecture is overkill for non-real-time use cases. If you're shipping:

• Audiobook narration (chapter rendered, file delivered, played back later)
• Podcast voice generation (episode batch-rendered, dropped into hosting)
• Video voice-over (rendered offline, mixed in post-production)
• E-learning courses (per-module rendered, packaged into SCORM)

...batch TTS is fine. It's typically priced lower per character (no streaming infrastructure overhead), produces equivalent audio quality, and integrates with existing post-production pipelines more cleanly than streaming.

The decision rule: if the user is going to be in a synchronous conversation with the audio, you need streaming. If they're going to listen to a pre-recorded file later, batch is fine.

The Hakim-specific stack

For teams building real-time voice on Hakim:

• Hakim Fast v1.3: streaming TTS, sub-120 ms TTFB at p95, autoregressive frame-level decoding, streaming HiFi-GAN vocoder.
• WebSocket endpoint: bidirectional, mid-stream cancel within 30-50 ms, persistent connection across cancel-resume cycles.
• HTTP/2 chunked: alternative transport for batch-adjacent use cases where mid-stream input updates aren't needed.
• Voice cache: voice metadata (timbre, prosody profile) loaded once per voice ID and reused across requests, so cold-start TTFB on a voice that's been used recently is similar to warm-path TTFB.

The streaming endpoint is documented at /docs/streaming; a working browser demo lives at /playground. If you want to benchmark Hakim's TTFB against your current TTS, /use-cases/ai-agents has a side-by-side latency calculator.

A short architecture checklist

If you're picking a streaming TTS for a real-time voice product:

1. Verify TTFB measurements are taken from request-send to first-audible-byte, not request-send to first-network-byte.
2. Test TTFB on 1-second and 5-second utterances; rule out chunked-batch dressed up as streaming.
3. Verify mid-stream cancel works and stops within 100 ms.
4. Verify the connection stays warm across cancel-resume cycles.
5. Match the buffer size on the client to the network's jitter profile; don't hard-code 200 ms.
6. Instrument both TTFB and chunk-to-chunk jitter in production; the second one is what causes user complaints when the first looks fine.

If you have an architecture question that doesn't fit one of these tests, email hello@tryhakim.ai with the use case and we'll route it to the engineering team.