---
title: "Voice Agent Barge-In: Hindi, Gujarati, English"
description: "Voice agent barge-in explained: stop playback, speech and the model reply, route Hindi, Gujarati and English, and test a booking agent before launch."
canonical: "https://buildwithjit.com/articles/voice-agent-barge-in-language-routing-guide"
date: "Nov 16, 2026"
updated: "2026-11-16T11:11:17.134826"
category: "Voice AI Agents"
tags: [Voice AI, Barge-In, Azure Speech, FastAPI WebSockets, Multilingual Agents]
---

# Voice Agent Barge-In: Hindi, Gujarati, English

## Key takeaways

- Voice agent barge-in means the caller can interrupt the agent while it is speaking, and the agent stops at once. Most voice agents feel broken without it, even when the answers are good.
- A real barge-in stops four things, not one: the audio playing in the browser, the speech still being generated, the language model reply still streaming, and the half-finished message in the chat history.
- In a Hindi, Gujarati and English agent, the speech recognition language and the voice language must be chosen together. A mismatch makes the agent sound wrong even when the text is right.
- Many "voice bugs" are ordinary front-end bugs. In my project, one round of fixes was React state being changed in place.

## What I built and why this guide exists

I built a voice agent called Dr. Alex that books medical appointments by voice. The backend is FastAPI with WebSockets, speech-to-text and text-to-speech come from Azure Cognitive Services, the language model is OpenAI GPT-4o-mini, and the front end is React with Tailwind. It handles four locales: en-IN, en-US, hi-IN and gu-IN.

During the work I rewrote the audio processing, the main WebSocket handler and the database layer, and then went through several bug-fix rounds. The ones that cost me the most time were barge-in handling, language routing and React state mutation bugs. This guide explains each one in plain terms, with a checklist you can use on your own voice agent. If you are still choosing your speech and model stack, read [the voice agent latency guide](/articles/voice-ai-agent-stt-llm-tts-latency-build-guide) first.

## What voice agent barge-in actually has to do

A human interrupts when they hear enough. The agent must treat that moment as an event, not as a new message that waits in a queue. When the caller starts speaking while the agent is talking, four things have to happen:

1. **Stop the playback in the browser.** Clear the audio queue and stop the current clip, not only the next one.
2. **Stop the speech generation.** If your server is still streaming synthesized audio, cancel it so old audio does not arrive after the interruption.
3. **Stop the model reply.** If the language model is still generating text, cancel that request. You are paying for tokens nobody will hear.
4. **Fix the history.** Save only what was actually spoken before the interruption, or mark the message as interrupted. If you save the full planned reply, the model believes the caller heard something they did not.

### The echo problem

If the agent's own voice comes back through the microphone, the agent can interrupt itself. Browsers can reduce this with the `echoCancellation` constraint on the microphone track, as described in [MDN's reference for echoCancellation](https://developer.mozilla.org/en-US/docs/Web/API/MediaTrackConstraints/echoCancellation). Turn it on, and test with speakers as well as headphones, because speakers are where it fails.

### Barge-in test checklist

| Test | What you should see |
|:-- |:-- |
| Interrupt in the first second of a reply | Audio stops immediately, no leftover words |
| Interrupt during a long reply | The next answer responds to the interruption, not to the old question |
| Stay silent after interrupting | The agent waits and does not resume the old reply |
| Play the agent through speakers | The agent does not interrupt itself |
| Check the saved history | It holds only the spoken part, or an interrupted marker |

## Language routing for Hindi, Gujarati and English

Language routing decides which language the agent listens for and which voice it speaks with. Getting it wrong is obvious to the caller. Hindi text read by an English voice, or an English answer to a Gujarati question, makes the agent feel careless.

### Choose recognition and voice together

Speech-to-text needs a locale to listen for, and text-to-speech needs a voice for the answer. Treat them as one setting, for example `hi-IN` for both, and change them together. Check which locales and voices your provider supports before you design the flow. For Azure, the [language support page](https://learn.microsoft.com/azure/ai-services/speech-service/language-support) lists them, and the [language identification guide](https://learn.microsoft.com/azure/ai-services/speech-service/language-identification) explains automatic detection and its limits.

### Practical rules I use

1. **Ask once, then remember.** At the start, offer the languages and store the choice for the session. Do not guess again on every turn.
2. **Do not flip on short words.** "Haan", "okay" and a name carry almost no language signal. A single short utterance should never change the session language.
3. **Let the caller switch on purpose.** Support a clear request such as "speak in Gujarati" and update both the listening language and the voice.
4. **Expect mixed speech.** Callers mix Hindi and English in one sentence. Pick one recognition locale per session and test it with mixed sentences, so you know how it behaves before callers find out.
5. **Tell the model the language.** Pass the session language in the system message, so the answer matches the voice.

## The React state bug that looked like a voice bug

Some of my "audio bugs" were front-end bugs. React expects you to replace state instead of changing it in place. If you push a new message into an existing array and set the same array again, the screen may not update, and the conversation view looks frozen even though the audio is fine. The React documentation explains the rule in [Updating Arrays in State](https://react.dev/learn/updating-arrays-in-state).

Two habits fix most of it:

- Create a new array or object for every state update.
- Keep fast-changing audio objects (the microphone stream, the socket, the playback queue) in refs, not in state, so they do not trigger re-renders on every chunk.

## Keep a booking agent safe

An appointment agent changes real calendars, so the model should not be trusted with details it can get wrong.

1. Read the details back before booking: name, date, time and language.
2. Create dates, times and booking IDs in code, not in the model's text. A model has no clock and will guess.
3. Keep personal details out of logs, and keep the agent to booking. If a caller asks a medical question, hand off to a person.
4. Add approval or a human handoff for anything risky, as described in [human-in-the-loop approval steps](/articles/human-in-the-loop-ai-approval-steps-automation-guide).
5. Watch cost per call. Interrupted replies that keep generating waste tokens, and [this cost control guide](/articles/how-to-control-ai-costs-in-automation-projects) shows how to set a budget.

## Before you launch: a ten minute test script

1. Interrupt the agent three times in one call.
2. Switch language in the middle of a call.
3. Say a mixed Hindi and English sentence.
4. Test on speakers and on headphones.
5. Book, change and cancel an appointment, and read the saved record.
6. Open the history and confirm it holds only what was spoken.

If you want a second pair of eyes on a voice agent project, you can reach me through [the contact page](/contact). I take on AI and automation work under JIGNESH IT SERVICE.

## FAQ

**What is barge-in in a voice agent?**
Barge-in is the caller's ability to interrupt the agent while it is speaking. The agent must stop the audio, cancel the pending reply and listen.

**Why does my voice agent keep talking after I interrupt?**
Usually only the playback was stopped. Queued audio, the speech stream or the model request is still running. Cancel all of them and clear the queue.

**Can one speech recognition setting handle Hindi, Gujarati and English?**
It depends on your provider and the feature you use. Check the supported locales and the language identification limits in its documentation, and test with real mixed sentences.

**Why does my agent interrupt itself?**
Its own voice is reaching the microphone. Enable echo cancellation, test on speakers, and add a short guard period right after the agent starts speaking.

**Should the language model decide the booking time?**
No. Let it collect the details, then validate and create dates and IDs in code and read them back to the caller.
