Case study · Architect · 35 min

Low-Latency Voice Agent

Design a phone/voice agent that keeps conversational latency under ~800ms turn-taking while calling tools.

voicelatencystreaming

Interview opener

Voice agent with tight latency budget end-to-end.

1.Where do you spend the latency budget first?
2.Streaming vs waiting for full tool results?
3.What do you cut when p95 blows the SLO?

Strong signals

Budget spreadsheet across STT/LLM/TTS/tools
Speculative or parallel tool calls
Graceful degrade to shorter replies

Hit 0/3

Weak signals (avoid)

Only 'use a faster model'
No budget allocation

Requirements

Channels: PSTN + webRTC; barge-in supported
Latency: user stop-talk → agent first audio < 800ms p95
Quality: WER < 8% English; task completion ≥ 70%
Scale: 200 concurrent calls; avg 4 min

Back-of-envelope Estimation

Metric	Calculation	Result
Concurrent calls	200	200
Audio streams	200 × 2	400
ASR cost	200 × 4 min × $0.006/min	~$4.80 / 4min burst
LLM tokens/call	~8K	1.6M tokens / 200 calls
Budget split	ASR 150ms + LLM 400ms + TTS 200ms	750ms
Tool calls	timeout 300ms or speak filler	UX critical

API Design

Endpoint	Method	Description
WS /v1/voice/session	WS	Bidirectional audio + events
POST /v1/voice/tools/result	POST	Async tool completion callback
POST /v1/voice/hangup	POST	End + store transcript

Data Model

Entity	Key fields	Notes
calls	call_id, from, started_at, outcome	CDR
utterances	call_id, role, text, t_ms	Timed transcript
latency_spans	call_id, stage, ms	ASR/LLM/TTS/tool

High-level Design

Streaming pipeline

Mic → VAD → streaming ASR (partials)
          → endpointing → LLM stream (token→TTS early)
          → streaming TTS → jitter buffer → speaker
Parallel: speculative tool prefetch on intent
Barge-in: cancel TTS + LLM on user speech
Edge colocated ASR/TTS with LLM region affinity

Deep Dive: Beating the 800ms Budget

Start TTS on first sentence tokens; don't wait for full answer
Use small specialized LLM for dialog; escalate hard tasks async
Filler audio ("one moment") only when tool > 400ms — measure annoyance
Colocate ASR→LLM→TTS in one region; avoid multi-hop oceans

Bottlenecks at Scale

Endpointing errors cause interruptions — tune VAD per accent
TTS GPU pool saturation — precompute common phrases
PSTN jitter — adaptive buffer 20–60ms

Failure Modes & Monitoring

SLO	Target	Alert
First audio p95	< 800ms	Page if > 1.2s
Barge-in success	> 95%	Warn
Call completion	≥ 70%	Weekly
Silent calls	< 0.5%	Page

AI & FDE Interview Room