A speech-to-text engine will hand your contact center software a full sentence even when the caller said nothing at all. If that transcript reaches a model holding tools, the agent can transfer a call, cancel an order or write to your CRM on the strength of words that were never spoken. The fix is not a smarter model. It is a plausibility check sitting between the transcriber and the thing that can act.
We found this the hard way while building the AudioSocket voice agent that runs under our own stack. Whisper is genuinely good at what it does. It is also, on near silence, a confident liar.
What a hallucinated transcript actually looks like
Whisper and models like it are trained to produce text. Given 120 milliseconds of hiss off a bad copper pair, they do not return an empty string. They return something. In our call logs the recurring offenders were end-of-video boilerplate: “Thank you for watching.” “Please subscribe.” Fragments of subtitle files that leaked into the training data and now surface whenever the audio carries no speech.
Read on a screen, those look harmless. On a phone call they are not. “Thank you” is a closing signal. A language model reading a conversation transcript treats it as one, replies politely, and winds the call down while the caller is still waiting for someone to pick up the thread.
Picture a caller on a train, phone held slightly away from her face while she digs out an account number. Three seconds of carriage rumble. The transcriber fills that gap with a pleasantry, the agent reads it as a goodbye, thanks her for calling and releases the channel. She was mid-sentence. She calls back, gets a different agent, and starts over. Your reporting shows two completed calls and no errors.
That last part is why this bug survives so long in production. It does not throw. It does not alert. It surfaces months later as a soft, unexplained dip in containment rate that everyone blames on the prompt.
OWASP moved Excessive Agency up to number three
The 2026 edition of the OWASP Top 10 for Large Language Model Applications puts Prompt Injection first and Sensitive Information Disclosure second. Third is Excessive Agency, up from sixth in the 2025 list. OWASP also published a separate Top 10 for Agentic Applications in December 2025, running from agent goal hijack at the top down to rogue agents at the bottom.
Excessive Agency is the risk that a model holds more permission to act than the reliability of its inputs justifies. Almost everything written about it assumes the attacker is a person. In telephony the attacker is usually just physics. A DTMF tone, a lawnmower outside an open window, a codec artefact on a weak mobile leg. None of that is adversarial. All of it produces the same result as a crafted prompt if the agent holds a transfer function and believes what it is handed.
My blunt view is that the industry spent 2025 arguing about jailbreaks and then shipped voice agents whose single most likely cause of a wrong action is a damp microphone.
Why voice is harder than chat
Text chat gives the model exactly what the user typed. No intermediate system is guessing. Voice inserts a second probabilistic model in front of the first, and the errors compound in one particular direction.
| Layer | Text chat | Voice call |
|---|---|---|
| Input fidelity | The exact characters typed | A second model’s best guess |
| Empty input | Empty string | Invented sentence |
| Confidence signal | Not needed | Not provided by most STT APIs |
| Cost of correcting | User retypes | Caller must talk over the agent |
The bottom row is the one people skip past. In chat a mistake is cheap, because the user simply types again. On a call, correcting the agent means interrupting it, which requires barge-in to work properly, which requires the voice activity detector to be tuned, which is the same subsystem that produced the problem. The recovery path runs straight back through the failure.
Four checks between the caller and a tool call
None of these involve the language model. They all live in the media path, and by the time the model is invoked the questionable transcripts are already gone.
A voice activity gate. An utterance does not open until the detector has seen 200 milliseconds of genuinely voiced audio. Below that threshold we never call the transcriber, so there is no text to hallucinate in the first place. Keypad taps, coughs and a chair scraping in an open plan office all die here.
A 300 millisecond lookback buffer. Gating on energy creates a problem of its own. A voice activity detector only calls a frame voiced once the talk spurt carries enough energy, so a quiet onset, a leading fricative or the closure before a plosive, has already been thrown away by the time the utterance opens. The transcript starts mid word, and the model quietly guesses at the rest. We keep the most recent unvoiced frames in a small ring buffer and prepend them when speech starts. The cost is fixed at 4.8 kilobytes per call, however long the call runs.
This one matters most immediately after barge-in, where the caller’s first word competes with the agent still speaking. It was suggested by a developer on the Asterisk subreddit, which tells you something about where the real edge cases get found.
A word density ceiling. Nobody speaks faster than about eight words per second, and that is already generous for an excited caller. We count the words in the returned transcript and divide by the audio we actually classified as voiced. Four words over 120 milliseconds implies 33 words per second, which is not a human being. Drop it. The whole check is one regular expression and one division.
One detail took us a second pass to get right. The prepended lookback audio deliberately does not count toward the voiced total. If it did, adding 300 milliseconds of quiet to every utterance would lift the ceiling by roughly two and a half words each time, which weakens the exact check the whole design depends on.
A known phrase blocklist. Some hallucinations sail through the density check because they are grammatically fine and long enough to look real. “Thanks for watching” over a full second of breathing is plausible on arithmetic. It is not plausible on a support line. A short list of training set boilerplate catches the ones that arithmetic cannot.
There is a fifth guard that is less obvious and bit us later. Some endpoints stop sending RTP entirely during silence, so Asterisk delivers no frames at all, and a silence counter driven by arriving frames simply freezes. The utterance never closes, and everything the caller said gets transcribed in one lump at hangup. We now treat any wall clock gap longer than 120 milliseconds between frames as silence in its own right.
What to ask before you buy
Almost every demo of contact center software with an AI voice agent is recorded in a quiet room on a good headset. That is precisely the condition under which none of this happens. Four questions are worth asking any vendor, including us.
- What happens to a transcript the STT returns over silence? If the answer is that the model handles it, there is no guard.
- Which actions can the agent take with no human in the loop? Transfer and hangup are usually fine. Refunds, cancellations and CRM writes deserve a confirmation turn.
- Is there a log line when a transcript gets rejected? You cannot tune a filter you cannot see, and the rejection rate is a useful signal about line quality.
- How is barge-in detected, and does the caller’s first word survive it? Ask for a recording made over a real mobile leg, not a headset.
Our own feature documentation answers these in writing rather than in a sales call, which is the standard we would want from anyone else. If you are comparing platforms more broadly, the writeup on open source Asterisk based contact center software covers where these systems tend to diverge, and the platform overview walks through the rest of the stack around the agent.
The part that generalises
You cannot make a speech-to-text engine stop inventing text. What you can do is make its output earn the right to trigger an action. Every check described here is arithmetic on numbers you already have: how long the audio was, how much of it was voiced, how many words came back. No second model, no latency worth measuring, nothing that needs retraining when your provider ships a new version.
The guards are open source and the behaviour they prevent is not specific to our platform. Anyone running Whisper in front of an agent that holds tools carries the same exposure, and can copy the same fix in an afternoon.
Frequently asked questions
Does using a better speech-to-text model fix hallucinated transcripts?
It reduces them, but it does not remove the failure mode. Every current STT model is trained to emit text rather than to abstain, so silence and noise still produce output. The plausibility check is cheap enough that there is no reason to drop it when you upgrade the transcriber.
Why not just ask the language model to ignore nonsense?
Because the model cannot tell the difference. A hallucinated “thank you” is a perfectly ordinary thing for a caller to say, and the model has no access to the underlying audio duration that makes it impossible. The check has to run where the timing information still exists, which is in the media path.
What is Excessive Agency in the OWASP Top 10?
It is the risk that an AI system holds more permission to act than its inputs justify: too many tools, too broad a scope, or no human confirmation on consequential actions. OWASP ranked it third in the 2026 Top 10 for LLM Applications, up from sixth the year before.
Does adding these checks slow the agent down?
No. The voice activity gate and the lookback buffer run on 20 millisecond frames as they arrive, and the density check is a word count on a string that already exists. The lookback buffer holds 4.8 kilobytes per call regardless of call length.
How would I know whether this is already happening on my calls?
Look for transcripts that are short, polite and unrelated to the surrounding conversation, particularly ones immediately followed by the agent closing the call. Then compare the transcript length against the duration of the audio segment it came from. If a four word sentence maps to a fraction of a second, that is a phantom.