Fine-Tuning IndicConformer for Nepali Speech Recognition

sttasrnepaliindicconformernemobenchmarking

नेपाली वाक् पहिचानको लागि एउटा नयाँ उत्कृष्ट मोडेल

We’ve been benchmarking Nepali speech-to-text systems for a while. This is the story of how a strong off-the-shelf model became an even stronger one — and the bug we had to find and fix to get there honestly.

Best result: CER 7.5% on FLEURS ne_np · Base model: AI4Bharat IndicConformer (120M, hybrid CTC-RNNT) · Fine-tune data: 158 hours of Nepali speech

Starting point: a strong model, used as-is

We’d already benchmarked seven Nepali STT systems — fine-tuned Whisper variants, wav2vec2, zero-shot baselines — on the FLEURS ne_np test set (726 utterances), all scored identically. Then we tried AI4Bharat’s IndicConformer, a 120M-parameter hybrid CTC-RNNT Conformer, completely off-the-shelf, no Nepali-specific tuning at all.

It won outright: 8.2% CER, beating the previous best fine-tuned Whisper (10.2%) while running 8x faster. A strong pretrained model beating a fleet of fine-tunes is worth noticing on its own.

Fine-tuning it further: two false starts

We didn’t stop there. We fine-tuned IndicConformer on 158 hours of Nepali speech (a small single-speaker corpus plus a large public one), expecting it to only get better.

It got worse, twice. A first fine-tune on a small 10.5-hour corpus pushed CER up to 10.9%. A second attempt on the full 158-hour corpus was worse still — 12.2%. Both times, the model’s own internal validation numbers looked great throughout training. Only the independent FLEURS benchmark told the truth.

Finding the actual bug

Looking at real transcriptions side by side, the fine-tuned model’s output was peppered with a literal <unk> token, almost always right where a comma or sentence break would naturally fall — something the base model never did.

The cause: IndicConformer’s tokenizer has no punctuation tokens at all — no danda (।), no comma, nothing. Our training transcripts had punctuation in them anyway. Every time the model saw one during training, the loss forced it toward an out-of-vocabulary fallback token. Repeated over tens of thousands of examples, it learned a bad habit — guess <unk> at every natural pause — and that habit leaked into every transcription it produced afterward, tripling the insertion-error rate.

The fix was simple once we knew what to look for: strip punctuation from the training transcripts (the same normalization we already use for scoring), so every training target is something the model’s vocabulary can actually represent. No new tokens, no architecture change — just cleaner training data.

The result, after the fix

Variant	CER %	WER %	Insertions
Zero-shot (base checkpoint)	8.2	24.2	314
Fine-tuned, punctuation left in (10.5h)	10.9	29.1	858
Fine-tuned, punctuation left in (158h)	12.2	31.4	1,040
Fine-tuned, punctuation stripped (158h)	7.5	23.4	267

Once the bug was fixed, the underlying fine-tuning benefit — which had been there all along, just hidden under the <unk> artifact — finally showed up: a real, reproducible improvement over the already-strong base model, with insertion errors dropping below what the base model produced.

What’s actually driving the improvement

Looking closer at individual examples, the gain isn’t spread evenly across everything. It’s concentrated in numbers:

Reference	Base model	Fine-tuned model
`802.11n मानक 2.4ghz...`	“आठ सय दुई प्वाइन्ट एक एक हन मानक दुई पोइन्ट चार गिगाहर्ज…”	“८०२१ यन मानक २४ गिगाहर्ज…”
`...15 अगस्ट 1940 मा...`	“…पन्ध्र अगस्ट उन्नाइस सय चालिसमा…”	“…१५ अगस्ट १९४० मा…”

The base model tends to fully verbalize numbers into spoken Nepali words. The fine-tuned model outputs literal digit characters instead — which happens to match how FLEURS’ own reference transcripts are written. Non-numeric content improved too, but more modestly. Worth knowing if you’re deciding whether this fine-tune matters for your specific use case.

Should we train it more?

We checked before assuming yes. Validation word-error-rate had already leveled off by the end of our two epochs — it even ticked up very slightly between the first and second — while training loss was still falling, but slower each epoch. That’s a plateau, not a runway. We’re not planning to just keep training longer; if we revisit this, we’d track FLEURS at each epoch rather than assume more training helps.

Get it

The fine-tuned checkpoint is public on Hugging Face at Oshara/indicconformer-ne-finetuned, along with a README covering the training setup, full benchmark numbers, and this same punctuation-bug story so anyone reproducing it can avoid the same trap.