A Nepali Voice for XTTS v2

ttsnepalixttsresearchevaluationvoice-cloning

नेपाली भाषाको लागि एउटा नयाँ आवाज

Zero-shot multilingual models like XTTS v2 can clone a voice from a few seconds of reference audio and speak in seventeen languages. Nepali isn’t one of them. This is the paper behind changing that: reusing an existing tokenizer instead of building a new one, and what that bought us on naturalness, intelligibility, and speaker fidelity.

Base model: XTTS v2 · New vocabulary: none · Training data: 10.5 hours · SCOREQ auto-MOS: 4.21

01 · The problem

Open Nepali TTS today is mostly limited to Piper and gTTS-class systems — serviceable, but not voice-cloning and not particularly natural-sounding. High-quality Nepali voices exist, but only behind commercial, closed APIs. Nobody had shown that a strong, self-hostable, voice-cloning Nepali system could be built cheaply on top of an existing multilingual model.

XTTS v2 already ships a Hindi language token and a Hindi text tokenizer. Nepali and Hindi are both written in Devanagari, and at the character level Hindi’s byte-pair vocabulary already covers Nepali text almost entirely.

So instead of training a new tokenizer or expanding the vocabulary — the usual approach when adding a language — we simply register ne as an alias that routes to the existing Hindi [hi] language token and text-cleaning path, and fine-tune only the autoregressive GPT decoder on Nepali audio. No new embeddings need to be learned from scratch, which avoids the typical cold-start problem of randomly-initialized tokens for a brand-new language.

We paired this with a Nepali number-and-date normalization front-end, so that digits, decimals, percentages, and Bikram Sambat dates (e.g. the year “2083”) are correctly expanded into spoken Nepali words rather than read digit by digit.


New vocab tokens	0
Training audio	10.5 h
Speakers	1
Released checkpoint	epoch 10

03 · How we measured it

We evaluated on NepTTS-Bench, a set of 205 phonetically controlled Nepali sentences spanning eight linguistic categories: minimal pairs, homographs, contrastive stress, question intonation, emotion, robustness, phrases, and newspaper-style ambiguities.

Four things were measured for every system in the comparison: automatic naturalness (SCOREQ predicted MOS), reference-free audio quality (TorchAudio-SQUIM), speaker similarity to the reference voice (SECS via WavLM), and round-trip intelligibility — synthesizing speech, transcribing it back with automatic speech recognition, and comparing to the original text.

The recognizer you use changes everything. Round-trip intelligibility is only as trustworthy as the speech recognizer doing the transcribing. Whisper-small, a common choice in prior Nepali TTS benchmarks, is itself a poor Nepali recognizer — its errors dominate the measurement, not the TTS system’s. So we fine-tuned our own Whisper-large-v3 for Nepali (11.5% character error rate on the FLEURS benchmark test set) and used it as the primary scoring instrument, alongside three other recognizers as cross-checks. Under the weakest recognizer, apparent errors were up to 4× inflated compared to our reliable one — the same audio, wildly different scores, depending purely on which recognizer was doing the listening.

04 · Results: the best open, self-hostable Nepali voice we could measure

Our fine-tuned system reaches a SCOREQ automatic-MOS of 4.21 — the highest among every open or self-hostable system tested, and third overall out of eleven systems, behind only two proprietary commercial voices. Speaker similarity to the target voice is strong (SECS 0.923).

Metric	Value
SCOREQ auto-MOS	4.206
SECS speaker similarity	0.923
intra-system consistency	0.966
SQUIM STOI / PESQ / SI-SDR	0.993 / 3.78 / 27.6 dB
SQUIM subjective MOS	4.69
F0 mean / variation	249.6 Hz / 3.58 semitones
Speaking rate	1.29 words/s

Measured with our reliable Nepali recognizer, round-trip character error rate is 8.8% — actually below that recognizer’s own error rate on natural, real-recorded Nepali speech. Our synthesized speech is at least as easy to transcribe as genuine human recordings, a strong signal of near-perfect intelligibility.

Naturalness and intelligibility usually trade off. Ours doesn’t have to.

Most TTS systems are good at one of the two: natural-sounding but hard to transcribe accurately, or highly intelligible but robotic. Compared against every other open, self-hostable Nepali system we could regenerate and score like-for-like — Edge-TTS, Piper, gTTS, and a LoRA-adapted Chatterbox model — our system is the only one that sits in the top tier on both axes at once: as intelligible as the most intelligible baseline (Edge-TTS), while scoring meaningfully more natural (4.21 vs. 3.6–3.7).

System	SCOREQ ↑	SQUIM-MOS ↑	CER ↓	WER ↓
Ours (xtts-ne)	4.206	4.69	0.088	0.312
Edge-TTS (Hemkala)	3.737	3.67	0.069	0.262
Edge-TTS (Sagar)	3.627	3.05	0.085	0.284
Piper	3.486	3.10	0.145	0.407
gTTS	3.023	3.14	0.236	0.480
Chatterbox-LoRA	3.994	4.69	0.484	0.674

Recognizer cross-check

Round-trip error for our system under all four recognizers used in the study — the spread shows exactly how much the choice of recognizer matters.

Recognizer	CER ↓	WER ↓
Whisper-ft (large-v3, ours)	0.088	0.312
XLS-R-Nepali	0.183	0.638
MMS-1b (npi)	0.199	0.487
Whisper-small	0.380	0.982

Per-category breakdown

NepTTS-Bench spans eight categories; results are sorted by error rate under our reliable recognizer. The benchmark is dominated by phonological minimal pairs (149 of 205 clips), so category-level results are more informative than the overall average alone.

Category (clips)	CER ↓	WER ↓
Newspaper ambiguities (4)	0.030	0.141
Emotion (8)	0.050	0.267
Phrases (10)	0.050	0.467
Phonological minimal pairs (149)	0.084	0.278
Contrastive stress (8)	0.110	0.448
Question intonation (10)	0.122	0.292
Robustness (10)	0.137	0.639
Homographs (6)	0.162	0.375
Overall (205)	0.088	0.312

What an 8.8% error rate actually sounds like. Across most categories, our fine-tuned recognizer’s transcription matches the target text exactly — a phonological minimal pair like “उसको काम राम्रो छ।” or a question like “खाना खानुभयो?” comes back character-for-character correct. The two interesting exceptions: a code-switched sentence mixing the Latin-script word “Facebook” with Devanagari is spoken correctly, but the recognizer transcribes the pronunciation phonetically in Devanagari script — a scoring artifact, not an actual synthesis failure, since the audio itself is intelligible. And we kept one genuine miss in the paper rather than hiding it, because honest error analysis matters more than a clean-looking table.

05 · A second architecture, for comparison

Beyond the open baselines, we also benchmarked a Nepali LoRA fine-tune of ResembleAI’s Chatterbox model — a different TTS architecture entirely — using the same reference voice and identical evaluation pipeline.

Because it clones the same reference speaker, its speaker similarity (SECS 0.935) is comparable to, if not slightly above, ours (0.923), and its naturalness scores are close to ours as well. But round-trip intelligibility is substantially worse — CER 0.484 vs. our 0.088, more than five times higher. LoRA-adapting a different backbone can match naturalness and speaker fidelity, but not speech-content accuracy, compared to full GPT-decoder fine-tuning on the Hindi-reuse path.

06 · What real listeners think

Alongside the automatic metrics, we ran a blind human listening study: native Nepali-speaking raters heard the same sentence rendered by five different systems, in randomized, blinded order, and rated the naturalness of each on a 5-point scale. Our system rated highest overall.

System	Human MOS [95% CI]	n ratings
Ours (xtts-ne)	3.761 [3.592, 3.930]	226
Piper	3.704 [3.571, 3.836]	226
Edge-TTS (Hemkala)	3.624 [3.481, 3.766]	226
Chatterbox-LoRA	3.261 [3.069, 3.453]	226
gTTS	2.655 [2.489, 2.820]	226

Matched paired comparison: all five systems rated on the same 226 trials by the same 16 native-Nepali listeners.

Humans and automatic metrics don’t always agree. The human ranking broadly agreed with the automatic SCOREQ ordering at the top and bottom of the field, but disagreed in the middle: Piper rated above Edge-TTS/Hemkala with human listeners despite the reverse under the automatic metric, and Chatterbox-LoRA, second-highest by SCOREQ, rated second-lowest with human raters. Automatic naturalness predictors and human listeners aren’t always sensitive to the same artifacts — which is exactly why a real human study is worth running, rather than trusting automatic scores alone.

07 · Why it works

Fine-tuning is what makes it intelligible. Reading Nepali text through XTTS’s stock Hindi path with no Nepali-specific fine-tuning at all is effectively unintelligible — a round-trip character error rate over 1.6, meaning more errors than characters in the transcription. Fine-tuning on just 10.5 hours of Nepali audio cuts that error rate roughly 18×, down to 8.8%. Script-sharing alone gets you nothing; the fine-tuning is doing the real work.

It’s specifically Hindi’s pretrained embeddings that make it cheap. To test whether reusing Hindi’s tokenizer was actually necessary — or whether any dedicated Nepali tokenizer would work just as well — we trained a brand-new Nepali byte-pair vocabulary from scratch on Nepali Wikipedia text, added it to the model with freshly (randomly) initialized embeddings, and fine-tuned under an otherwise identical recipe. Naturalness and speaker similarity came out statistically indistinguishable between the two versions — but intelligibility was clearly worse with the cold-start vocabulary (CER 0.117 vs. 0.088). The saving isn’t just about sharing a script; it’s that Hindi’s tokens arrive already pretrained, and a from-scratch vocabulary of similar size simply needs more than 10.5 hours of data to catch up.

08 · Takeaway

Reusing an existing, related language’s tokenizer — rather than expanding vocabulary from scratch — lets a multilingual TTS model learn a genuinely strong new-language voice from a small amount of data. The same recipe should extend to other Devanagari languages that currently lack dedicated text-to-speech support, like Maithili and Bhojpuri.

We’re releasing the training and evaluation code, the fine-tuned checkpoint, and per-utterance benchmark results, so the approach can be verified and built on directly.