Streaming Voice Turn Detection and Backpressure: Endpointing, Interruptions, and Audio Queue Management

The most common production failure in voice assistants is not "hearing nothing at all," but incorrect turn boundaries: a user's turn is committed early when they pause to breathe, the system takes too long to respond after they finish speaking, a fragment of old TTS resurfaces after an interruption, or the browser accumulates PCM without limit when the network slows and eventually treats audio from ten seconds ago as real-time input. VAD can only determine whether a frame resembles human speech; it cannot decide on its own when a turn is ready to commit.

Voice and Audio AI Applications introduced ASR, TTS, VAD, and interruption handling. This article brings them together as an observable state machine and backpressure contract. The transport-layer event and recovery principles can also be combined with LLM Streaming Protocols, but audio cannot be discarded or reordered arbitrarily like text tokens.

Give Every Audio Chunk a Deterministic Identity

The session negotiates the codec, sample rate, channel count, frame duration, and protocol version. Every chunk carries a monotonic sequence, capture time, sample count, and turn generation. The server rejects duplicate, regressing, and cross-generation chunks, and reconstructs media time from the actual sample count rather than relying on unstable network arrival times.

audio: {encoding: pcm_s16le, sampleRateHz: 16000, channels: 1, frameMs: 20}
runtime:
  gatewayImage: registry.example/voice-gateway@sha256:88a0...
  vadModelDigest: sha256:2f13...
  asrModelRevision: asr-prod-2026-06
  providerApiContractSha256: 510b...
turn:
  speechStartFrames: 3
  preRollMs: 240
  silenceEndMs: 650
  maxTurnMs: 30000
  endpointGraceMs: 250
buffer:
  clientHighWaterMs: 1000
  serverHardLimitMs: 2000
  resumeBelowMs: 400

These parameters are bound to the VAD, ASR, language, device type, and echo-cancellation configuration; one "optimal 650 ms value" cannot be shared across the entire site. The client clock is used only for diagnostics, while the server session clock controls timeouts. Reconnection carries the last acknowledged sequence and generation. If continuity cannot be proven, a new turn begins instead of forcibly splicing two audio segments together.

Combine Acoustic and Semantic Endpointing with a State Machine

The state set must include at least idle, possible_speech, speaking, possible_end, committed, and cancelled. The state moves from possible_speech to speaking only after several consecutive speech frames, while restoring the pre-roll ring buffer to avoid clipping the first word. Brief silence moves only to possible_end; if speech resumes, the state returns to speaking. A unique turn_id is generated and committed only after reaching the silence threshold or maximum turn duration, or after an explicit button press ends the turn.

An ASR interim result is a replaceable view, not a business fact. Every revision carries a revision and replacement range. Only a final transcript confirmed after endpointing may trigger business tools, write operations, or task-success billing. Network bytes, received audio milliseconds, ASR/model tokens, and GPU time are metered separately from the moment resources are accepted. Cancellation also records actual consumption; the absence of a final result does not mean there was no cost. If a sentence is clearly incomplete, endpointing may be extended once within the hard limit. State transitions can follow the Agent Runtime State Machine, while deterministic code continues to guard side effects.

Backpressure Must Bound "Audio Age"

A browser captures audio at a rate dictated by real time, while downstream processing may slow because of ASR, network, or model queues. The queue is measured in both milliseconds and bytes, and exposes desiredSize or explicit credit. After reaching the high-water mark, downstream processing must continue to drain at its available rate while upstream credit is set to zero, pausing or rejecting new enqueue operations. Pausing downstream processing would only make the backlog worse. Once the server reaches the hard limit, it returns a classifiable overload error instead of continuing to accept data and pretending it is real-time.

PCM already inside an active turn cannot have frames randomly dropped from its middle. Brief congestion uses a bounded buffer. If the microphone cannot pause and the backlog cannot be expected to drain before the hard deadline, cancel the entire uncommitted generation, meter the resources already consumed, ask the user to repeat themselves, and resume with a new sequence. Only old silence from before speech or refetchable derived data may be discarded, and every gap must be recorded. Web Streams backpressure propagates consumption capacity; it cannot stop a microphone from producing real-world audio.

Monitor queue duration in milliseconds, the age of the oldest frame, credit exhaustion, gaps, duplicates and out-of-order chunks, endpointing latency, and resource-release latency after cancellation. WebSocket bufferedAmount alone is insufficient: it only reports bytes at one sender and cannot explain the server-side ASR queue or media duration. Cancellation propagation from Next.js AI Streaming Response Management applies to the control plane, but the audio and playback generations must also be cleared here.

Barge-in Is an Atomic Generation Switch

When sustained user speech is detected during AI playback, first use echo cancellation and a short confirmation window to rule out speaker feedback. Then atomically increment the generation: stop local playback, cancel the remaining TTS, cancel or freeze the old LLM, close the old ASR turn, and submit the new user audio with pre-roll. All late chunks from the old TTS, text deltas, and tool candidates are discarded because their generation no longer matches.

If the old response has already triggered an irreversible tool, an interruption cannot undo that fact; the system can only enter a compensation flow or explain the situation to the user. Tool side effects must therefore never be triggered directly by an interim transcript. The playback side records the exact audio position that was actually played. Context for the next turn contains only content the user heard or content the system explicitly marks as unplayed, preventing the model from assuming that the entire response was delivered.

Release with Noise and Congestion Replays

The golden set covers short utterances, long pauses, filler words, coughing, background speech, overlapping speakers, Bluetooth jitter, echoes, weak networks, server pauses, and repeated interruptions. Store the audio hash, expected start and end windows, final text, and allowed revisions; do not place real user recordings directly in the test repository. Release gates can require zero first-word clipping, zero missed critical turns, endpointing P95 within target, zero playback from an incorrect generation, audio output stopping within 500 ms after cancellation, and queues never exceeding the hard limit.

New VAD or endpointing parameters first run in shadow mode and must not simultaneously drive user-visible responses. Then roll them out by device and language at 5%, 20%, and 50%. If early commits, tail latency, repeat-request rates, or queue drops cross their thresholds, routing reverts to the previous configuration digest. New generations use the old state machine, while existing turns finish under their pinned versions. A rollback cannot merely change the silence duration; it must also restore the VAD, ASR finalization rules, buffering, and interruption strategy together.

Reliable turn detection does not merely mean that conversations work most of the time. It means that every audio chunk, endpointing decision, backpressure discard, and generation cancellation can be explained. Under overload, an explicit retry is preferable to disguising stale audio as real-time conversation.

Further Reading

• Voice and Audio AI Applications
• LLM Streaming Protocols
• Agent Runtime State Machine
• Next.js AI Streaming Response Management