Practical Voice/Audio AI Applications

Over the past two years, voice AI has quietly crossed the threshold from "it can roughly understand you" to "it can be turned into a product." Real-time meeting transcription, low-latency conversational voice assistants, AI dubbing, and real-time cross-border translation still felt like demos in 2024, but today they can run in production. Voice and text LLMs present very different engineering challenges: latency is not merely a matter of a few hundred milliseconds; within that window lies the perceptual dividing line between responsive and sluggish. A one-word error cannot be fixed by simply "regenerating"; the user has already heard the awkward half-sentence.

This article breaks voice AI applications into five areas: the capability stack, product scenarios, engineering considerations, architecture choices, and real-world pitfalls. It explains the actual challenges involved in moving from a demo to production.

1. The Voice AI Capability Stack

Voice applications are primarily assembled from three atomic capabilities:

• ASR (Automatic Speech Recognition, speech-to-text): Converts an audio stream into text. Representative options include OpenAI Whisper, Deepgram, AssemblyAI, and real-time recognition services from Alibaba Cloud and Tencent Cloud.
• TTS (Text-to-Speech): Synthesizes text into natural-sounding speech. Representative options include ElevenLabs, Azure Neural TTS, OpenAI TTS, and music-grade TTS from MiniMax and ByteDance.
• End-to-end voice assistants: Audio goes directly in and audio comes directly out. The model natively understands speech, including emotion and tone, without an explicit text intermediate. GPT-4o realtime voice and Gemini Live follow this approach.

Two other capabilities are often overlooked but determine the quality of the experience: VAD (Voice Activity Detection), which determines when the user starts and finishes speaking, and wake-word / interruption detection, which determines when the assistant should stop talking.

Most applications are orchestrations of these components. Understand the latency, cost, and failure modes of each component, and you understand the entire product.

2. Typical Product Scenarios

Different scenarios have dramatically different requirements for latency, accuracy, and naturalness. You must classify the scenario before selecting technology:

A common mistake is applying the metrics for a "conversational voice assistant" to "dubbing generation." You can regenerate dubbing ten times and choose the best version. For a conversational assistant, every additional 300ms of latency makes the experience feel noticeably worse.

3. Streaming ASR: Producing Text as the User Speaks

Non-streaming ASR, where a complete recording is uploaded before results are returned, works for offline scenarios. Meeting transcription, captions, and voice conversations require streaming: text must begin appearing before the user has finished speaking.

Streaming ASR normally uses WebSocket or bidirectional streaming HTTP. The client continuously sends audio chunks, with PCM 16kHz/16bit serving as the de facto standard, while the server returns incremental results (interim) and final results (final) as it receives them:

// 客户端:采集麦克风并推流到 Deepgram
const ws = new WebSocket(
  "wss://api.deepgram.com/v1/listen?encoding=linear16&sample_rate=16000&interim_results=true&endpointing=300",
  { headers: { Authorization: `Token ${DG_KEY}` } }
);

const mic = await navigator.mediaDevices.getUserMedia({ audio: true });
const track = mic.getAudioTracks()[0];
// WebRTC 的 captureStream + AudioWorklet 把 PCM 切成 ~100ms 帧
const node = audioCtx.createAudioWorkletFromModule("pcm-processor");
node.port.onmessage = (e) => ws.send(e.data); // ArrayBuffer

ws.onmessage = (ev) => {
  const { type, channel, is_final } = JSON.parse(ev.data);
  const text = channel.alternatives[0].transcript;
  if (is_final) commitToTranscript(text);   // 落库 / 推到 LLM
  else renderInterim(text);                 // 实时打字效果
};

Several engineering considerations matter:

• Chunk size: Frames of around 100ms balance latency and throughput. Larger frames feel sluggish, while protocol headers consume too much bandwidth with smaller frames.
• Silence endpointing: The server usually treats 300-500ms of silence as the end of an utterance. Set it too short and coughing or breathing will split the sentence; set it too long and users in a conversational scenario will not tolerate the wait.
• final does not mean absolutely correct: A streaming final result may still be revised based on later context, especially punctuation and proper nouns. Design an updatable event model for transcripts rather than an append-only model.
• Audio format negotiation: Opus saves bandwidth on mobile devices, while PCM gives the server lossless decoding. Make sure frontend encoding and server-side decoding agree.

4. Low-Latency TTS: Time to First Audio Determines Perceived Speed

The key TTS metric is not "how long does it take to synthesize this passage?" It is TTFB (Time To First Byte, or time to first audio), the interval between the user pressing play and hearing the first syllable. For a five-second utterance, a three-second TTFB followed by streaming audio provides a much better experience than waiting 1.5 seconds for the entire clip to be returned.

Nearly all modern TTS services support streaming: text chunks are supplied incrementally, PCM/MP3 frames are pushed as they are synthesized, and the client plays them while downloading. The key is prioritizing the first sentence. Synthesize and play the first short sentence, or even half a sentence, immediately while later sentences are synthesized concurrently during playback.

// OpenAI TTS 流式调用,首音延迟可控
const res = await fetch("https://api.openai.com/v1/audio/speech", {
  method: "POST",
  headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "gpt-4o-mini-tts",
    voice: "alloy",
    input: longText,
    response_format: "mp3",
    stream: true,
  }),
});

const player = new AudioContext();
const src = player.createMediaElementSource(new Audio());
// reader 边读 chunk 边喂给 MSE / AudioBufferSource,实现边下边播
for await (const chunk of res.body) appendToBuffer(chunk);

Practical recommendations:

• Send sentences to TTS as separate chunks instead of waiting for the complete LLM output. Streaming LLM output and streaming TTS synthesis can be pipelined, making time to first audio roughly equal to "the time for the LLM to produce its first sentence + TTS TTFB."
• Pre-generate common phrases. Synthesize fillers such as "Okay," "Let me check," and "Please wait" in advance, then play one immediately after the user speaks to mask the actual backend latency. This is the open secret behind the responsiveness of commercial voice assistants.
• Voice consistency: Lock the same voice ID and version for a session. A TTS vendor's model upgrade may change the voice.
• Prosody breaks down on long text: Passages longer than 200 Chinese characters can sound flat. Split them by paragraph and then concatenate them, taking care with pauses and intonation at the joins.

5. VAD and Interruption Handling

In conversational scenarios, the real challenge is often not ASR or TTS itself, but when to start listening and when to stop speaking.

VAD (Voice Activity Detection) sits at the entry point. Streaming to ASR should begin as soon as the user starts speaking, rather than after recording is complete. Lightweight VAD systems such as Silero VAD and WebRTC VAD can classify each 30ms frame as "speech/no speech" on the device, providing low latency and better privacy. The critical VAD parameters are start-point triggering, which prevents the first word from being clipped, and the end-point silence threshold, which prevents a breath from being mistaken for the end of an utterance.

The harder problem is barge-in (interruption): the user starts speaking while the AI is talking. The handling flow is:

TTS 播放中 ──VAD 检测到人声──> 立即停止 TTS 播放 + 取消剩余合成
                              └─> 把当前 ASR 文本送进 LLM,开始新一轮回复

Pitfalls include:

• Acoustic echo cancellation (AEC): When audio plays through speakers, the microphone can pick up the AI's own voice and mistake it for a user interruption. You must use AEC, either built into WebRTC or provided at the system level, or strictly require headphones.
• Half-second latency window: Allow a 150-300ms confirmation window after VAD triggers to avoid false activation from environmental noise or coughing.
• Cancellation must be complete: Cancel the streaming LLM request, cancel remaining TTS chunks, and clear the playback queue. Otherwise, another half-sentence may play after the interruption.
• Interruption semantics: An interruption may not mean "stop"; the user might be adding information. Provide both the incomplete ASR text from the interruption and the unplayed AI text as context, then let the LLM decide whether to stop and answer again or continue.

6. Multimodal Voice Assistant Architecture: Cascaded vs. End-to-End

There are two approaches to conversational voice assistants:

A. Cascaded pipeline (ASR → LLM → TTS): Each stage can be selected, optimized, and replaced independently. The architecture is clear, observable, and suitable for gradual rollout. The cost is cumulative latency: ASR endpointing waits 300ms, the LLM takes 500ms to produce its first token, and TTS has a 300ms TTFB. Together they easily exceed 1.2 seconds and feel sluggish.

[麦克风] → VAD → ASR(流式) → LLM(流式) → TTS(流式) → [扬声器]
                  │                          │
                  └──── barge-in 取消线 ─────┘

B. End-to-end speech models (GPT-4o realtime, Gemini Live, and others): The model natively consumes audio tokens and emits audio tokens without an explicit text intermediate. Advantages include low latency, potentially 300-500ms, and the ability to preserve subtle distinctions in tone, emotion, and interruptions. It can even detect that "the user is laughing." The disadvantages are higher cost, less control because safety filtering cannot be applied at the text layer, harder debugging because there is no intermediate text to log, and continued sensitivity to noise and multiple speakers.

Practical recommendation: start with a cascaded pipeline, then adopt end-to-end models after it works. The intermediate text produced by a cascaded architecture is invaluable. You can insert safety filtering, function calling, RAG, and audit logging, all of which are difficult with end-to-end models. End-to-end models suit scenarios such as companionship or emotionally expressive conversations, where the intermediate text representation has little value and latency is extremely important.

Hybrid architectures are also common. An end-to-end model manages the conversational flow, while critical decision points such as placing orders or querying a database use an explicit ASR → LLM → tool → TTS pipeline, combining low latency with control.

7. Real-World Pitfalls

• Cumulative latency: In the worst case, a three-stage pipeline waits for ASR endpointing, processes a long LLM prompt, and then performs TTS synthesis, easily reaching three seconds. Always evaluate P95/P99 rather than average latency. Pipelining and pre-synthesized fillers are essential.
• Accents and noise: WER can reach 5% on clean read-speech datasets but jump directly to 20%+ in meetings with background music or customer-service recordings with regional accents. Always test ASR in your actual acoustic environment rather than trusting a vendor demo.
• Emotion and tone: TTS handling of emotions such as happiness and anger is far less mature than text LLMs. The same "Okay" should sound different when spoken angrily or happily, but most TTS systems cannot do this naturally. Where emotion matters, use SSML annotations or specialized emotional TTS, or simply have the LLM include explicit discourse markers in the text.
• Multiple speakers (speaker diarization): Identifying "who said what" is essential in meetings, but streaming diarization remains unreliable and usually requires a second offline pass. Perform only "segmentation" in real time and assign speakers afterward.
• Privacy and compliance: Uploading recordings to the cloud triggers GDPR, China's Personal Information Protection Law, and industry-specific requirements in fields such as healthcare and finance. Either run Whisper-small on the device or provide explicit notice, encrypt data locally, and use a short retention period. Voice data in enterprise settings is often more sensitive than text because it may contain biometric information.
• Hallucinated ASR: Whisper may "invent" text during silent segments, with "Thanks for watching!" being a well-known hallucination. In streaming scenarios, VAD must remove silent segments before they are sent to ASR instead of streaming indiscriminately.

8. Tool Selection Reference

• ASR: For real-time English, use Deepgram for extremely low latency or AssemblyAI for strong diarization. For Chinese, use real-time recognition from Alibaba Cloud or Tencent Cloud. For offline or privacy-sensitive workloads, use self-hosted Whisper; faster-whisper is the de facto standard and is four times faster than the original.
• TTS: ElevenLabs sets the quality ceiling, particularly for emotion and cloning. Azure Neural TTS and Google offer enterprise stability. For Chinese voices, consider MiniMax, ByteDance's Volcano Engine, or Alibaba CosyVoice. For local/openai-compatible deployments, use Kokoro / Mellotron.
• Real-time voice APIs: OpenAI Realtime API (GPT-4o), Deepgram Voice Agent, LiveKit Agents (an open-source orchestration framework that can connect providers), and Vapi / Retell (conversational phone agent platforms).
• VAD: Silero VAD (on-device, accurate, lightweight), WebRTC VAD (extremely fast but coarse), and Deepgram endpointing (server-side).
• Orchestration: Assemble your own system with WebSocket, or use a voice Agent framework such as LiveKit Agents / Pipecat. These frameworks already handle interruptions, VAD, and pipeline orchestration, potentially saving one or two months.

When selecting technology, do not look only at model leaderboards. Examine three things: P95 latency curves, WER/naturalness on your actual audio, and whether pricing hides an "audio duration vs. character count" trap.

Summary

• Voice AI = ASR + TTS + VAD + orchestration. Product quality is determined by latency and interruption handling, not the metrics of a single model.
• Streaming is the minimum requirement: ASR must produce text while the user speaks, TTS must play audio while generating it, and LLM streaming must be pipelined with TTS. Time to first audio affects perceived responsiveness more than total latency.
• Start conversational assistants with a cascaded ASR → LLM → TTS pipeline. Use the intermediate text for safety, tools, and auditing, then adopt end-to-end speech models where extremely low latency is required.
• Three practical pitfalls: echo causes false interruptions and requires AEC; Whisper hallucinates during silent segments and requires VAD first; and three-stage pipelines accumulate latency and require pre-synthesized fillers plus pipelining.
• Do not select tools based on leaderboards; test actual audio. WER can differ by a factor of four between clean datasets and noisy recordings.

In one sentence: the engineering challenge for text LLMs is "is it correct?" The engineering challenge for voice AI is "is it fast?" And speed always means perceived speed, not benchmark speed.