Running speech-to-text locally: what whisper.cpp gets wrong when nobody is talking

I dictate into my own tools all day, and I did not want a transcript of my working life sitting on somebody else's server. The Web Speech API would have shipped every word to Google. So the whole thing runs on the laptop through whisper.cpp, and it works well enough that I now talk to my software more than I type at it.

Getting there took three bugs that nobody warns you about.

Batch transcription feels broken even when it is correct

The obvious build records the whole utterance, then transcribes it. That is wrong in a way you feel immediately: the screen stays empty while you talk, and the longer your sentence, the longer the wait.

My first version was worse than that. Every 1.1 seconds it re-sent the entire accumulated recording, so it re-transcribed from the first syllable on every pass. Latency grew with the length of the sentence.

whisper-stream fixes this properly. It keeps a sliding window in memory and re-decodes only the last few seconds:

--step 400 --length 4000 --keep 300

Text lands about 400ms behind your voice and never gets slower. On an M4 the cold start is 0.26s for base.en and 0.35s for small.en, cheap enough to spawn per keypress, so the microphone is never held open between utterances.

Use small.en. It costs 90 milliseconds more to load and it is meaningfully better on proper nouns, which is most of what I dictate. base.en heard a repository name as "Valhalla the true V."

The model does not know when you have stopped talking

This is the one that will bite you. Whisper does not output silence during silence. It outputs fluent, confident, completely invented sentences.

Left running in a quiet room, mine produced:

Nobody in their right mind but everything I myself approach.

The recovery existed before the diagnosis did.

Neither of those was said by anyone. The -vth voice-activity flag looks like the answer and is not: outside VAD mode whisper-stream ignores it entirely.

What worked was gating on the browser side. The level meter already computed RMS from the audio stream, so that same number became the speech gate. Anything decoded more than 1.5 seconds after the last real sound gets discarded before it reaches the field.

Two smaller failures came out of the same window:

The decoder loops when starved. I caught it emitting the token YOU two hundred times in one four-second window. A window that short cannot legitimately hold that, so a repetition check drops it.

The overlap arrives twice. Consecutive windows overlap by 300ms so a word straddling the boundary is not clipped in half. The consequence is that the words inside that overlap get decoded twice: draft a reply to the followed by to the carrier composes as "to the to the carrier." Stitching on the longest repeated run fixes it.

The gate must decide what to accept, never what to erase

I wrote the echo gate as one line:

tail.current = hot ? data : ""

Recorded messages then started vanishing at the moment of sending, and only sometimes, which made it look like a race condition.

It was not. whisper-stream commits a segment roughly every 3.6 seconds, and everything before that commit lives in a live tail. Any sentence shorter than 3.6 seconds existed only as that tail. The moment I stopped talking and the hot window lapsed, the next frame overwrote the whole thing with an empty string, about a second before I pressed send. Long sentences partly survived because some of them had already been committed. Short ones died completely.

The gate now decides what to accept. It never erases what it already heard.

Do not run voice detection on requestAnimationFrame

The silence clock read its RMS inside a requestAnimationFrame loop, which is what you reach for when the same number is driving a level meter.

rAF is a rendering hook. Browsers stop firing it when the window is hidden, occluded or in a background tab. With the clock frozen, every decoded segment failed the "was he actually speaking" test, all of them were discarded, and the end-of-utterance timer never fired. Switch windows and dictation went permanently deaf.

It samples on a 40ms interval now. Voice detection cannot depend on anything being drawn.

What it looks like working

Spoken into the microphone, captured from the endpoint:

Show me the
Show me the uncommitted
Show me the uncommitted file.
Show me the uncommitted files and then draw.
Show me the uncommitted files and then draft a re-
Show me the uncommitted files and then draft a reply to the

Final composed text: Show me the uncommitted files and then draft a reply to the carrier.

That is a 465MB model, no API key, no network, and nothing leaving the machine. Three of the four bugs above were about detecting the end of a thought rather than about transcribing words, so budget your time accordingly.