Building a voice AI agent: the speech-to-text, LLM and text-to-speech pipeline

The pipeline sounds simple.

Speech goes in, text comes out, a model decides what to say, and a voice reads the answer back. Most builders sketch it as three boxes and a couple of arrows, then get surprised when the demo works and the live version doesn't. The three components are not the hard part. What breaks agencies is everything that happens at the boundaries between them: latency stacking up across three separate API calls, context getting lost between turns, and a business process underneath that was never designed for a phone conversation in the first place.

The latency budget nobody sets

Each hop in the pipeline adds delay. Transcription has to wait for a pause in speech before it commits to text. The model has to receive that text, reason over it, and generate a reply. The voice engine has to synthesize audio from that reply before anything plays back. Chain three services from three different vendors and the gaps compound in ways that don't show up until the workflow is actually connected end to end.

One builder had a voice agent that scheduled appointments through a messaging platform, and the round trip from spoken question to spoken answer ran to 45 seconds. Nobody had budgeted latency as a design constraint. It was the sum of three components each built to be correct, not fast, and nobody owned the total.

The fix is treating end-to-end response time as a number to protect from day one, the same way you'd protect a budget or a deadline, rather than something to notice after a client complains. That means testing the full round trip early, not testing each service in isolation and assuming they'll compose cleanly.

Language and dialect are not solved by picking a good model

The pipeline's English performance tells you almost nothing about how it will handle another language, and this catches people who've only ever tested in their own tongue. Speech recognition, the underlying model, and the voice synthesis were all trained overwhelmingly on English data, so accuracy degrades unevenly as you move away from it.

One agency had a voice agent performing well in English but stumbling on Arabic, and the gap ran across all three layers: recognition missed regional pronunciation, the model lost some conversational nuance in translation, and the synthesized voice sounded foreign enough that callers noticed immediately. That combination was enough to put a real deal at risk. If a client's market runs in anything other than English, that has to be tested as its own project phase, not assumed to inherit from the English build.

State is the part that quietly breaks first

A voice agent isn't just answering one question. It's tracking where a caller is in a multi-step process: have they given their email yet, is the appointment actually booked, should the workflow move to the next stage. This is where systems most often go wrong in ways that are hard to spot in a quick test. An agent might keep asking for information it already has, because the call's internal status and the workflow's stage tracking have drifted out of sync. The caller experiences this as the agent not listening. The real cause is usually two systems disagreeing about what state the conversation is in, and the fix is auditing how call status maps to pipeline stage rather than patching the prompt.

Build the plumbing before the personality

It's tempting to spend early effort on how the agent sounds and what it says, because that's the visible, demoable part. The plumbing underneath, on the other hand, decides whether any of that matters: telephony that reliably connects calls without getting flagged as spam, a CRM that updates in step with the conversation, and a scheduling system that reflects reality a second after the call ends. A single missed sync on any of those and the caller has a bad experience regardless of how natural the voice sounds.

The one thing to hold onto is this: a voice pipeline is judged on its weakest connection, not its strongest component. Build and test the seams before you polish the voice.