Teaching XTTS v2 to Speak Nepali
XTTS v2 लाई नेपाली बोल्न सिकाउँदा
A production voice model shipped with 17 languages, but no Nepali. This is how we fine-tuned it into a fluent Nepali narrator: how the model works, how we chose the checkpoint that generalises best, and the text-pipeline fixes that stopped it from babbling.
Update. Since this was first published we’ve added three things: a head-to-head comparison against a completely different TTS model to see if our approach is the only way to get here, a controlled test of why borrowing from Hindi works so well, and results from an ongoing public listening test where real people rate the audio. Jump to sections 07–09.
Base: XTTS v2 (Coqui TTS) · Method: full GPT fine-tune · Data: 4,140 clips · Lang: ne · नेपाली
01 · The gap: a great voice model that could not say नमस्ते
XTTS v2 is a zero-shot voice-cloning model: give it six seconds of any voice and a line of text, and it speaks that line in that voice. It ships fluent in seventeen languages.
Nepali is not one of them. The closest neighbour is Hindi, which shares the
Devanagari script and much of the phonology, so the honest baseline is to feed Nepali
text to the model while telling it language="hi" and hoping for the best. It is
passable, but the vowel lengths drift, distinctly Nepali sounds get Hindi-fied, and the
prosody sits in an uncanny valley for a native ear. We wanted a model that actually
speaks Nepali, so we fine-tuned one.
| 4,140 | 18th | 0.41× | 24 kHz |
|---|---|---|---|
| Nepali clips | supported language | real-time factor | output audio |
02 · How the model works
The model never turns letters straight into sound. First it writes your sentence out as “sheet music”, a string of audio notes, then a fast band plays that music back in your voice. Teaching it Nepali just means coaching the songwriter; the band does not change.
Under the hood, four components pass the work down a line. Tap any stage to see what it does, or hit trace to watch an example sentence flow all the way through, and flip the toggle between a plain-English and a technical explanation.
You hand the model a short clip of any voice, about six seconds. It listens once and captures what makes that voice unique: its tone, pitch and texture.
A Perceiver resampler encodes the reference audio into speaker latents (gpt_cond_latent) plus a fixed speaker_embedding voice fingerprint. Zero-shot, with no per-speaker training.
reference in🎙️ 8-second voice clip
↓
[ voice fingerprint ]Your Nepali sentence is chopped into small subword pieces and stamped with a “this is Nepali” tag, so the model knows which sounds to expect.
A BPE tokenizer (the base text vocabulary, reused untouched) splits Devanagari into subwords, prefixed with a language token. Nepali rides the existing [hi] route.
text → tokensनमस्ते
↓
[ne] न | म | स्तेThis is the songwriter. Reading the voice and the words together, it writes the sentence out as a sequence of “audio notes”, one note at a time, like predicting the next word in a text. This is the only part we retrained for Nepali.
A GPT-2 autoregressive transformer consumes [speaker latents + text tokens] and samples DVAE audio tokens one step at a time. The full GPT is what we fine-tuned; a broken or empty prompt makes it free-run into babble.
generates1024, 812, 45,
991, 33, … , 1025
(start … stop)Those notes are not random numbers; they come from a fixed “alphabet” of about a thousand little sound-pieces the model learned during pre-training.
The DVAE (discrete VAE) defines a 1,026-entry audio codebook. Token 1024 = start, 1025 = stop; the other 1,024 are the vocabulary the GPT predicts in. Script-level, not language-level.
the codebook1,026 audio tokens
1024 = start
1025 = stopFinally the band plays the sheet music: a fast decoder turns the notes into a real sound wave, speaking your text, in the voice you gave it.
A HiFi-GAN vocoder maps the GPT’s hidden latents + speaker embedding to a 24 kHz waveform. No diffusion, so decoding is fast, roughly 0.41× real-time on GPU.
waveform outhidden latents
↓
🔊 24 kHz audioTwo facts here explain almost everything downstream. First, the GPT is an autoregressive sampler: feed it a broken or empty prompt and it drifts into nonsense, just like a chatbot rambling. Second, “adding a language” does not require new audio machinery: the DVAE, the vocoder and the tokenizer work at the script level, not the language level. We only have to teach the GPT what Nepali sounds like.
Paper: XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model (Casanova et al., Interspeech 2024). The architecture roots in Tortoise-TTS.
03 · The fine-tune: adding ne without touching the vocabulary
Because Nepali and Hindi share Devanagari, we did not need a new tokenizer or new audio
tokens. The trick was small: register a new language code ne in the model config,
and route it through the existing Hindi token and text cleaners. From there it is a
standard XTTS GPT fine-tune, restored from the base weights.
The data
A single-speaker Nepali corpus, converted from an LJSpeech-style layout into Coqui’s three-column transcript format (audio, text, speaker) and split 95 / 5 (seed 42): 3,933 train / 207 eval, 4,140 total. Input 22,050 Hz, output 24,000 Hz.
Training configuration
Full GPT fine-tune (not frozen), restored from the base checkpoint, on the Coqui
GPTTrainer. Roughly 1,966 steps per epoch, batch size 2 × grad-accum 2, AdamW
(betas 0.9 / 0.96, weight decay 1e-2), learning rate 5e-6 with a MultiStepLR ×0.5
schedule, Perceiver resampler enabled.
Checkpoint selection
Save every 10 epochs, then pick the one that generalises best from the eval curve rather than the last one written.
The learning rate is deliberately conservative: we are nudging a pretrained model, not training from scratch.
04 · Why epoch 10, not epoch 30
We saved a checkpoint every 10 epochs and watched the eval loss. It splits into two very different stories. The text cross-entropy (did the model pick the right tokens for this script) is essentially solved almost immediately and stays flat around 0.028. The mel cross-entropy, the actual acoustic quality, bottoms out early and then climbs steadily as training continues. That rising eval curve is textbook overfitting to a single speaker.
■ eval mel-CE (rising = overfitting)■ eval text-CE ≈ 0.028 (solved, flat)
The takeaway. Later checkpoints memorised the training speaker better but generalised worse. Epoch 10 is the sweet spot: enough to sound natively Nepali, not so much that it overfits. So the epoch-10 checkpoint is what we ship, even though later ones exist.
05 · Listen: base model vs. fine-tune
The identical Nepali line, same reference voice. First, stock XTTS v2 driven as Hindi; then our epoch-10 Nepali fine-tune.
Spoken text
नेपालका हिमालहरू संसारभर प्रसिद्ध छन्। यहाँको प्राकृतिक सौन्दर्य अतुलनीय छ।
Spoken text
नेपालका हिमालहरू संसारभर प्रसिद्ध छन्। यहाँको प्राकृतिक सौन्दर्य अतुलनीय छ।
Numbers, read like a newsreader
A real remittance news paragraph, dense with Nepali numerals and large figures (खर्ब, अर्ब, करोड), run through the fine-tune with the full text pipeline. Digits are expanded to Nepali words before synthesis, and decimals are read as दशमलव (१०.६ → “das dashamlab cha”).
Spoken text
यस्तै, २०८३ वैशाखमा मात्र २ खर्ब ५७ अर्ब ४९ करोड रुपैयाँ रेमिट्यान्स भित्रिएको छ । यो रकम मासिक रूपमा हालसम्मकै बढी हो । यसअघि चैतमा २ खर्ब ९ अर्ब रुपैयाँ रेमिट्यान्स भित्रिएको थियो । गत वर्षको वैशाखमा रेमिट्यान्स आप्रवाह १ खर्ब ६५ अर्ब ३० करोड थियो । वैशाखमा अमेरिकी डलरमा रेमिट्यान्स आप्रवाह ३३ प्रतिशतले वृद्धि भई १३ अर्ब २६ करोड पुगेको छ । गत वर्ष यस्तो आप्रवाह १०.६ प्रतिशतले बढेको थियो ।
06 · How it measures up
We evaluated the epoch-10 model on NepTTS-Bench (205 phonologically-designed sentences) and added the voice-cloning and prosody metrics that a read-speech benchmark does not cover. Two stories emerge: intelligibility and quality are among the best on the benchmark, and the one soft spot is pitch expressiveness.
Intelligibility and quality
ASR round-trip character error rate (lower is better) and quality MOS (higher is better), ranked against the benchmark’s systems.
| Metric | Score | Rank |
|---|---|---|
| Whisper round-trip CER | 0.380 | 2nd of 11 |
| MMS-1b round-trip CER | 0.199 | 2nd of 10 |
| XLS-R Nepali CER | 0.183 | 5th of 10 |
| SCOREQ auto-MOS | 4.21 | 3rd of 11 |
Voice cloning, prosody, and signal quality
Computed on all 205 clips: speaker similarity to the reference voice (WavLM-SV), pitch and prosody (librosa), and reference-free signal quality (torchaudio SQUIM).
| Metric | Value | Read |
|---|---|---|
| Speaker similarity (SECS) | 0.923 | high cloning fidelity (same-speaker threshold ≈ 0.86) |
| Voice consistency across clips | 0.966 | one stable identity, no drift |
| SQUIM MOS (estimate) | 4.69 | reference-free naturalness |
| SQUIM STOI / PESQ / SI-SDR | 0.99 / 3.78 / 27.6 dB | clean signal, low distortion |
| Pitch expressiveness (F0 std) | 3.58 st | vs. 4.95 st for natural speech: flatter intonation |
The one soft spot. On every intelligibility and quality metric the fine-tune ranks at or near the top, and it clones the reference voice with high fidelity (SECS 0.923). The clearest gap is prosody: its pitch varies about 30% less than natural Nepali speech, so the intonation is a little flatter. That, not clarity or voice identity, is the next thing to improve.
The weights are published at Oshara/xtts-v2-nepali (epoch 10 recommended, epoch 20 included).
07 · Is a full fine-tune actually necessary?
Full GPT fine-tuning is not the cheapest way to adapt a TTS model. A popular alternative is LoRA: instead of updating the whole network, you bolt on a small set of extra weights and train only those, which is much lighter on compute. So we asked the obvious question: could a lightweight adapter on a different voice-cloning model get us just as far?
We took Chatterbox, an open multilingual TTS model from Resemble AI, and applied the same idea, a Nepali LoRA adapter on top of its Hindi checkpoint, cloning the exact same reference voice we use for XTTS. Then we ran it through the identical evaluation pipeline.
| Metric | Our XTTS fine-tune | Chatterbox + LoRA adapter |
|---|---|---|
| Round-trip CER (lower is better) | 0.088 | 0.484 |
| Round-trip WER (lower is better) | 0.312 | 0.674 |
| SCOREQ auto-MOS | 4.21 | 3.99 |
| Speaker similarity (SECS) | 0.923 | 0.935 |
The two land close together on how natural they sound and how faithfully they clone the reference voice, and after tuning Chatterbox’s own generation settings the gap in speaker similarity essentially disappears. Where they still pull apart is whether you can actually understand the words: our full fine-tune’s round-trip error rate is roughly half the LoRA-adapted alternative’s. A lightweight adapter can get a different architecture to sound like a fluent, natural-sounding Nepali voice, but it does not yet get it to reliably say the right words as often. For that, the deeper, full fine-tune still wins.
08 · Is it just the alphabet, or something deeper?
Our whole approach rests on one bet: Nepali is close enough to Hindi that we can reuse Hindi’s existing knowledge instead of starting from zero. But “close enough” could mean two very different things. Maybe it is purely that they share the Devanagari script, so any model that already reads Devanagari gets a head start. Or maybe it goes deeper: the model that was pretrained on Hindi actually learned something about the sounds, not just the letters, and that is the part we are borrowing.
To find out, we ran a controlled test: we trained a second version of the model with a brand-new Nepali-only vocabulary, built entirely from scratch, using the exact same data and recipe, instead of routing through the existing Hindi tokenizer. If script-sharing were the whole story, the from-scratch version should do about as well. It did not, it came out measurably less intelligible.
The takeaway. It is not the shared alphabet doing the heavy lifting, it is the pretrained Hindi knowledge underneath it. Reusing Hindi’s language route is not a shortcut around Devanagari, it is a shortcut around having to teach the model what these sounds are like in the first place, which is exactly why it is so cheap to do.
09 · What real listeners think
Automatic metrics are a proxy. To check whether they agree with actual human ears, we built a small blind listening test: on each screen, a listener hears the same sentence rendered by several systems, presented in random order with no labels, and rates how natural each one sounds on a 1–5 scale.
With more than 500 ratings collected so far across the self-hostable systems tested, the ranking lines up with what the automatic metrics predicted:
| System | Average listener rating (1–5) |
|---|---|
| Our XTTS fine-tune | 3.95 |
| Microsoft Edge-TTS (Nepali voice) | 3.75 |
| Piper (open-source, offline) | 3.48 |
| Google Translate’s Nepali voice | 2.52 |
The Chatterbox-LoRA comparison from Section 07 was added to the same listening test more recently and is still collecting ratings, so we are holding off on a number for it until there is enough data to trust. We also recently swapped in a freshly regenerated batch of Chatterbox samples after tuning its generation settings, which is what drove the intelligibility improvement in the table above, so its listening-test number will reflect that improved version going forward.
10 · Why it was babbling between sentences
A good checkpoint is not enough. In production the model would sometimes lapse into gibberish between sentences, and the cause was never the weights. It was the text we fed the autoregressive GPT. Three culprits, all invisible in a log:
- Zero-width characters. ZWNJ / ZWJ (U+200C / U+200D) are rampant in Devanagari text and sail straight into the tokenizer, derailing it.
- Empty and punctuation-only chunks. Splitting on the danda
।can yield a chunk that is just।or whitespace, an empty prompt, so the GPT free-runs into noise. - No clear stop. Chunks without terminal punctuation, plus a default high sampling temperature, let the decoder run on.
The fix: normalise first. Strip zero-width characters, collapse whitespace, drop
empty and punctuation-only chunks, and guarantee a terminal । on every chunk. Then
generate at temperature 0.65 with repetition penalty 5.0, and insert 150 ms of silence
between sentences. Clean speech, no babble.
Numbers and decimals
Digits are expanded to Nepali words before anything else, so a decimal point never gets mistaken for a sentence boundary. The point becomes दशमलव and the fractional part is read digit-by-digit:
| Input | Spoken as |
|---|---|
| तापक्रम 36.6 डिग्री | …छत्तीस दशमलव छ… |
| मूल्य 3.14 रुपैयाँ | …तीन दशमलव एक चार… |
| १००.०५ रुपैयाँ | एक सय दशमलव शून्य पाँच |
11 · Where it landed
- Fluent Nepali prosody from a 4k-clip single-speaker fine-tune, with no vocabulary
or architecture changes, just a new
nelanguage route over the Hindi token. - Epoch 10 ships, chosen from the eval curve rather than the last checkpoint.
- Held up against a different architecture. A LoRA-adapted Chatterbox model gets close on naturalness and voice cloning, but our full fine-tune is still clearly ahead on getting the words right.
- The Hindi-reuse trick is not just about the alphabet. A controlled test confirms it is Hindi’s pretrained knowledge of the sounds, not merely the shared script, that makes this approach cheap.
- Backed by real listeners, not just metrics. An ongoing blind listening test with 500+ ratings currently places it ahead of every other self-hostable Nepali voice we compared it against.
- Robust long-form text. Zero-width scrubbing, sane sentence splitting, inter-sentence silence, and Nepali number and decimal expansion make it safe on real articles.
- Fast. 0.41× real-time on GPU via the HiFi-GAN decoder.
Next: a blind epoch-10 versus epoch-20 listening test, growing the Chatterbox-LoRA comparison to a full sample size, and tuning temperature and chunk length for very long paragraphs.
Get it
The weights are on the Hugging Face Hub at Oshara/xtts-v2-nepali (epoch 10 recommended, epoch 20 included), and the training, inference, and evaluation code is at github.com/OsharaAI/xtts. Inference pulls the model straight from the Hub, so a few lines get you speaking Nepali in any cloned voice.
XTTS v2 · Nepali fine-tune · epoch 10 · © 2026 Oshara AI