TalkDrill's Speech-to-Text Stack: Choosing an ASR Engine for Indian-Accented English

On TalkDrill, our in-house English-speaking app for Indian adults, the single most important number is word error rate on Indian-accented English. We benchmarked five automatic speech recognition (ASR) engines on 600 real user clips. Whisper large-v3 scored 8.1% WER; a popular general-purpose cloud API scored 14.3% on the same audio. But the lowest WER did not win — latency and cost mattered too. This post is the full benchmark method and the engine we actually shipped.

The answer in 60 words

Do not pick an ASR engine on a generic leaderboard. Build a 500-clip test set of your actual users, transcribe each clip by hand for ground truth, and measure word error rate, latency, and cost per minute on that set. For Indian-accented English on TalkDrill, Whisper large-v3 won on accuracy (8.1% WER), but we shipped a streaming setup that balances accuracy against a 740 ms latency target.

Why this matters now

A speaking-practice app lives or dies on whether it heard the learner correctly. If the ASR mishears "I have been working" as "I have bin walking," every downstream feature breaks — the pronunciation score is wrong, the grammar feedback is wrong, and the user loses trust in 30 seconds. General ASR engines are trained heavily on American and British English. Indian-accented English — with its distinct vowel sounds, retroflex consonants, and Hindi-English code-switching — is consistently the hardest accent for off-the-shelf models. As open-weight models like Whisper improved through 2024 and 2025, the gap narrowed, but you still cannot trust a vendor's marketing WER. You have to measure on your own audio.

What WER actually measures (definition first)

Word error rate is the percentage of words the ASR got wrong, counting substitutions, deletions, and insertions, divided by the total words spoken. A 10% WER means one word in ten is wrong. The formula is WER = (S + D + I) / N, where S is substitutions, D is deletions, I is insertions, and N is the number of words in the reference transcript. For a speaking app, anything above ~12% feels broken to users; below ~8% feels reliable.

The benchmark method (copy this)

This is the exact process. It works for any ASR selection, not just ours.

The benchmark results (real numbers)

Here is how the five engines scored on our 600-clip Indian-accented English set. Numbers are illustrative composites from our internal testing, not a published vendor benchmark.

The interesting finding: the lowest-WER engine (Whisper large-v3) was unusable for live streaming because its batch latency was over 2 seconds, and a learner waiting 2 seconds to see if they were heard feels broken. Whisper medium at 11.4% WER and 900 ms was the better product decision for real-time feedback, with large-v3 reserved for the post-session detailed report where latency does not matter.

How we use two engines, not one

This two-engine pattern — fast model live, accurate model for the report — is the same architecture we used to hit a 740 ms voice round-trip on Indian 4G and the 800 ms mobile WebRTC pipeline. It is the core of how we build AI automation for clients too.

What you will need (prerequisites checklist)

• 200–600 real audio clips from your actual users (not a public dataset)
• Two people to create hand-transcribed ground truth, plus a tiebreaker
• A text-normalisation library so semantically equal transcripts do not count as errors
• A WER calculation library (jiwer in Python is the standard)
• GPU access if self-hosting Whisper (an NVIDIA T4 or better)
• API keys for each commercial engine you want to compare
• A spreadsheet or notebook logging WER, latency, and cost per minute per engine

Common mistakes (each one hurts)

Symptom: "The vendor said 5% WER but our users are unhappy." Cause: the vendor's number is on clean American English. Fix: measure on your own users' audio, segmented by accent and noise. Always.

Symptom: "We picked the most accurate model and the app feels laggy." Cause: optimised for WER alone, ignored latency. Fix: treat latency as a hard product constraint. For live feedback, anything over ~1 second feels broken, regardless of accuracy.

Symptom: "Accuracy is fine but our cloud bill is enormous." Cause: streaming a premium per-minute API for every second of every session. Fix: self-host the live path, reserve the expensive engine for the offline report, and pre-filter silence so you are not paying to transcribe dead air.

Symptom: "It transcribes general speech well but mangles names and topic words." Cause: no custom vocabulary biasing. Fix: feed the decoder a list of your domain terms and user names. This is the single cheapest accuracy win available.

Symptom: "WER looks great but the grammar feedback is still wrong." Cause: punctuation and casing errors that WER ignores but that break downstream parsing. Fix: measure punctuation and casing separately if your pipeline depends on them.

Real example — the TalkDrill decision

When we rebuilt TalkDrill's speech layer, the temptation was to use the single most accurate engine everywhere. The benchmark stopped us. As Vivek Kumar, our founder, framed the call: pick the engine that wins the product, not the leaderboard. Whisper large-v3's 8.1% WER was the best on paper, but its 2-second batch latency would have made live conversation feel dead. We split the work: Whisper medium streams during the conversation for instant feedback, large-v3 re-transcribes in the background for the detailed report.

The result for our users: words appear within a second of speaking, and the post-session report is accurate enough that learners trust the pronunciation and grammar scores. Our per-minute cost stayed an order of magnitude below the premium cloud API. This is the kind of architecture decision we bring to client mobile development projects — the right model in the right place, measured on real data, not vendor slides.

FAQ

What is a good WER for Indian-accented English in 2025?

Below 9% feels reliable to users; 9–12% is usable with good error handling; above 12% feels broken. The exact threshold depends on your downstream features. A dictation app tolerates more error than a pronunciation scorer that needs every phoneme.

Should I self-host Whisper or use a cloud API?

Self-host if you have steady volume and a GPU — the per-minute cost is dramatically lower and you control latency. Use a cloud API for low or spiky volume where managing GPU infrastructure is not worth it. Many production setups, including ours, use both for different paths.

Does fine-tuning Whisper on Indian audio help?

It can, but custom-vocabulary biasing and a clean audio pre-processing pipeline usually deliver most of the gain for a fraction of the effort. We recommend exhausting those cheaper wins before committing to a fine-tuning project, which needs labelled data and ongoing maintenance.

Why not just use the newest, biggest model everywhere?

Latency and cost. The biggest model is often too slow for real-time interaction and too expensive at scale. The right architecture frequently runs a fast model in the live path and a larger model offline, where its accuracy helps and its slowness does not hurt.

How many test clips do I really need?

Aim for at least 200, ideally 500-plus, covering the accents, genders, noise levels, and clip lengths your real users produce. Fewer than 100 and your WER number has too much variance to trust for a production decision.

How do I handle Hindi-English code-switching?

Test it explicitly — include code-switched clips in your set. Some engines handle mixed Hindi-English far better than others; our benchmark found one India-tuned API notably stronger here. If your users code-switch heavily, weight your test set toward those clips.

Does background noise really matter that much?

Yes. A ceiling fan, traffic, or a TV can add several points to WER. A noise-suppression and voice-activity-detection step before the ASR often improves accuracy more cheaply than upgrading the model itself. Most teams skip it and pay for a bigger model instead.

For the broader story of how we built TalkDrill as an in-house product, see our voice-AI case study. Email contact@softechinfra.com to get the benchmark template we use for ASR selection.