How I Built a Production Voice AI Agent: STT, LLM and TTS Latency, Step by Step

Most voice AI demos look great for thirty seconds. Then someone calls with a bad connection, talks for a minute without a pause, or asks something that is not in the script, and the whole thing falls apart.

I have spent the last few months building a voice agent for BytezTech called Aria. It listens, searches our knowledge base, answers in a natural voice, and writes a summary after the call. This post is the honest build log: the stack I picked, where the time goes, and the bugs that cost me the most time.

If you are building something similar, I hope this saves you a few evenings.

What the agent actually does

Aria handles one conversation turn at a time. The flow is simple to describe:

1. The caller speaks and the audio is captured.
2. Speech-to-text (STT) turns the audio into text.
3. The text goes to the language model, along with relevant facts pulled from our knowledge base.
4. The model's reply goes to text-to-speech (TTS).
5. The caller hears the audio, and the cycle repeats.
6. When the call ends, a separate step analyses the full conversation.

That is the whole idea. The hard part is not any single step. It is that the caller feels the sum of all of them as one silence before the agent answers.

The stack I chose

I did not pick these because they top a leaderboard. I picked them because they worked for my users: callers who mix Hindi, Gujarati and English in one sentence. If your callers speak only American English, your best choices may be different.

Where the time goes

This is the part most tutorials skip. A voice agent feels natural only when the gap between the caller finishing a sentence and hearing the first word back stays short. People start to feel uncomfortable once that pause grows beyond a second or so.

The delay is made of five parts added together:

• Sending the audio to your server
• Speech-to-text
• Searching the knowledge base
• Writing the reply with the language model
• Turning the reply into speech

I am not going to paste a neat table of averages here, because the numbers change with the caller's network, the server region and how busy each provider is on that day. A number that was true on my laptop will be wrong on yours. What matters more is knowing how to measure it for your own setup, and what pattern to expect.

How I measure it

Do not trust a stopwatch in your head. I add a timestamp before and after every step and save them with the session. In n8n that is one small Code node that writes Date.now() around each call. After a day of test calls I had real averages for my setup and, more importantly, I could see the slowest turns, which are the ones callers remember.

The pattern I noticed

The language model is usually blamed first, but on a fast inference provider like Groq it is rarely the biggest delay. The steps that grow with audio length, speech-to-text and text-to-speech, are where long turns get slow. A one-line answer is cheap. A long, detailed answer spoken in full is not.

That is why I keep the agent's replies short and let the caller ask for more. It is a design choice that helps the pause and sounds more like a real receptionist.

Three problems that nearly broke it

1. The 30-second audio limit

Sarvam's speech-to-text accepts audio only up to a fixed length, around thirty seconds. In testing I always spoke in short sentences, so I never noticed. The first real caller talked for almost a minute and got an error.

The fix was to cut long recordings into smaller pieces, send each piece for transcription, and join the text back in order. Cutting in the middle of a word hurts accuracy, so I split on silence where I could.

2. Merging the audio chunks back together

The same limit works the other way. Long replies have to be spoken in pieces, which means you get several audio files back and need to play them as one. Simply gluing the bytes together does not work for WAV files, because every file carries its own header. The result was a click, or worse, a file that stopped after the first piece.

What worked was reading the raw audio frames from each piece and writing them into a single new WAV file:

import wave

def merge_wavs(paths, output_path):
    with wave.open(paths[0], "rb") as first:
        params = first.getparams()

    with wave.open(output_path, "wb") as out:
        out.setparams(params)
        for path in paths:
            with wave.open(path, "rb") as piece:
                out.writeframes(piece.readframes(piece.getnframes()))

This only works if all pieces share the same sample rate and channel count, so I keep the TTS settings identical for every piece in a reply.

3. Two ways to start the same workflow

I wanted to test from a simple web form, but real calls arrive as webhooks. At first I built two copies of the workflow, and they slowly drifted apart. One had a fix, the other did not.

I rebuilt it so both the form and the webhook feed the same flow, and a single early step turns the input into one format. Now there is one place to fix a bug. This sounds small, but it saved me the most maintenance time of anything on this list.

Why I moved off Twilio and Airtable

My first version used Twilio for calls and Airtable to store sessions. It worked, but two things annoyed me. Every phone call added carrier cost, and Airtable became slow and awkward once I was saving every turn of every conversation.

I moved the voice side to WebRTC, so callers talk through the browser, and moved session storage to Notion. For my case this lowered the per-call cost and made the data easier for the rest of the team to read. It is not a perfect answer. If you need real phone numbers that people can dial from any mobile, you will still need a telephony provider.

Post-call analysis: the part clients actually like

The call itself is only half the value. After a conversation ends, a separate step reads the whole transcript and produces a short summary, the caller's intent, and whether a human needs to follow up.

In practice, this is what business owners ask about first. They do not want to listen to recordings. They want to open one page in the morning and see what happened overnight.

What I would do differently

• Measure from day one. I started timing steps only after callers said the agent felt slow.
• Test with messy audio. Quiet studio recordings hide almost every real problem. Test with traffic noise, a speakerphone and a fast talker.
• Keep one input path. Every extra entry point is another place for bugs to hide.
• Plan for failure. If speech-to-text fails, the agent should say "Sorry, could you repeat that?" instead of staying silent. Silence on a call feels like a crash.

What is next

The current design handles one turn at a time. The next big improvement is streaming: starting to speak while the model is still writing the rest of its answer. That alone can remove a large part of the pause. I will write about it once I have tested it properly.

Final thoughts

A good voice agent is not about picking the most famous model. It is about the boring details: audio limits, timing, fallbacks and one clean way in. Get those right and the agent starts to feel like a helpful receptionist instead of a demo.

If you are building a voice agent, or planning one for your business, you can reach me through the contact page. I am happy to talk through your setup.