Teaching a Machine to Speak Nepali

मेसिनलाई नेपाली बोल्न सिकाउँदा

A from-the-trenches account of fine-tuning Chatterbox v3, a voice-cloning text-to-speech model, for Nepali. The tokenizer bugs, the phantom sentences, the overfitting hunt, and the audio that came out the other side.

Base: Chatterbox v3 (Hindi) · Method: LoRA · rank 32 · Data: 4,138 clips · Lang: ne · नेपाली

01 · The mission — one model, any voice, in Nepali

Give it a few seconds of someone’s voice and a line of Nepali text, and it speaks that line in their voice. No per-speaker training. That is the goal.

Chatterbox is an open multilingual TTS model from Resemble AI. The v3 family ships a dedicated Hindi checkpoint, and Hindi shares the Devanagari script and most of its phoneme inventory with Nepali. That overlap is the whole bet. Rather than train a Nepali model from scratch, we graft Nepali onto the Hindi model with a small, cheap LoRA adapter and a corpus of roughly 4,000 Nepali recordings.

It sounds clean on paper. In practice, the path was a chain of subtle bugs, each one invisible until you actually listened to the output. This is that story.

02 · Under the hood — how the sound gets made

Text does not become audio in one leap. It flows through a pipeline, and every bug in this post lives at one of these boundaries: text not tokenizing right, tokens not conditioning right, audio not stitching right.

We only fine-tune the T3 transformer, and even then only a LoRA adapter injected into its attention and MLP projections. The vocoder (S3Gen) and voice encoder stay frozen. That keeps the trainable footprint tiny: about 210 small matrices instead of a two-billion-parameter rewrite.

TARGET_MODULES = ["q_proj", "v_proj", "k_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]
LORA_RANK    = 32
LORA_ALPHA   = 64
LORA_DROPOUT = 0.05

03 · The first bug — the tokenizer that spoke in fragments

Every generation starts by telling the model what language it is speaking, using a single special token like [hi] for Hindi. So we passed [ne] for Nepali. Reasonable. Except [ne] was never in the vocabulary.

The fix was to register [ne] as a proper single special token in the tokenizer, mirroring how [hi] (ID 722) was defined. It landed at ID 2111:

# before: [ne] -> [1706, 1712, 1720]  (garbage)
# after:  [ne] -> [2111]              (one clean token)

But fixing the language tag uncovered a deeper problem in the preprocessing.

def nepali_normalize(text: str) -> str:
    text = unicodedata.normalize("NFC", text)        # NOT NFKD
    text = re.sub(r"([0-9०-९]+)\.([0-9०-९]+)",
                  r"\1 दशमलव \2", text)                # 12.5 -> 12 दशमलव 5
    # standardize anusvara/chandrabindu + vowel length
    return text

We verified every fix before training: [ne] is one token, decimals expand, chandrabindu survives.

[ne] ID: 2111
sample encode (25 tokens): [1706, 1712, 1720, 1740, 1702, 1734, 7, 2]...
decimal test decode: '१२ दशमलव ५ किलो'
chandrabindu test (8 tokens): OK
Tokenizer: all checks passed

04 · The phantom sentence — the danda, the decimal, and a sentence that was not there

TTS models degrade on long inputs, so long paragraphs get split into chunks, synthesized separately, and stitched back together. Nepali ends sentences with a danda, ।, not a period. Simple plan: replace । with . and split there.

_DANDA_SENTINEL = '\x00'
 
def split_sentences(text: str) -> list:
    text = text.replace('।', _DANDA_SENTINEL)                       # danda to sentinel
    parts = re.split(rf'(?<=[{_DANDA_SENTINEL}?!])\s*', text)
    chunks = [p.strip() for p in parts if p.strip()]
    return [c.replace(_DANDA_SENTINEL, '.') for c in chunks]        # restore

Sentences still longer than about 40 seconds get sub-chunked at commas and pause markers, and any fragment below a minimum length gets merged back into its neighbour, because the model repeats and hallucinates when handed a two-word stub with no context.

05 · The smaller demons — the seed that never seeded

A reproducibility bug hiding in plain sight. The seed defaulted to 0, and the guard was:

if args.seed:          # 0 is falsy in Python, so set_seed() never runs
    set_seed(args.seed)

06 · The key insight — do not start Nepali from noise. Start it from Hindi.

After adding [ne] to the tokenizer, its embedding row existed in the model, but it was random noise. We measured it against the trained [hi] embedding:

[ne] embedding norm:        0.385   # random init
[hi] embedding norm:        0.553   # trained, meaningful
[ne] vs [hi] cosine sim:    0.058   # essentially orthogonal
[ne] vs random token sim:   0.041   # it IS just noise

NE_ID, HI_ID = 2111, 722
with torch.no_grad():
    model.t3.text_emb.weight.data[NE_ID] = \
        model.t3.text_emb.weight.data[HI_ID].clone()   # warm start
 
# keep the embedding table trainable at a gentler LR
model.t3.text_emb.weight.requires_grad_(True)
optimizer = AdamW([
    {"params": lora_params},
    {"params": [model.t3.text_emb.weight], "lr": args.lr * 0.1},
], lr=args.lr)

It is the language-model equivalent of telling a Spanish speaker to learn Portuguese. You do not hand them a blank slate, you hand them everything they already know that transfers.

07 · The investigation — why was it overfitting so fast?

An earlier run seemed to overfit almost immediately. The instinct was “too little data,” but the real culprit was the validation set itself.

The picture is clean. Validation loss falls steadily and bottoms out around epoch 10 (val 3.2016), then begins a slow drift upward while training loss keeps falling. That divergence is the textbook overfitting signal, and it tells us exactly which checkpoint to keep. Epoch 10 is the model.

08 · Listen for yourself — the audio

Every bug above was found by ear. Here is what each stage actually sounds like. Start with the reference, the voice we clone from a single short clip.

The model we shipped

The epoch-10 checkpoint: best validation loss, cleanest output.

Long-form, chunked and stitched

A full multi-paragraph article run through the sentence splitter, silence trimmer and stitcher: the payoff of the chunking work in section 04.

09 · The reference trap — a longer reference clip makes it worse, not better

It is tempting to feed the cloner a long sample of the target voice, on the theory that more audio means a better clone. The opposite is true. The model only looks at the beginning of the reference and discards the rest:

ENC_COND_LEN = 6  * S3_SR      # T3 speech-token prompt: first 6 seconds
DEC_COND_LEN = 10 * S3GEN_SR   # S3Gen decoder reference: first 10 seconds

Hand it a 30-second clip and roughly 20 seconds are thrown away. Worse, if those first few seconds happen to hold a breath, a pause, or a non-representative phrase, that is the entire basis for the cloned voice. A long clip also tends to start with lead-in audio that is the least clean part of the recording.

The same text and model, changing only the reference clip:

One more detail surfaced here. The dateline dash in काठमाडौँ — नेपाल was being voiced as a sound. A dash is a visual separator, not a phoneme, so the normalizer now converts every dash (em, en, and double-hyphen) into a comma-length pause before tokenizing.

10 · What’s next — the next experiment

The shipped model fine-tunes on the Hindi base. But there is a second bet worth testing: the generic 23-language multilingual base. It has seen far more diverse scripts and prosody, so it may generalize better to a new language, even if it is less Devanagari-specialized than the Hindi checkpoint.

The training script is built and waiting. Same LoRA recipe, same tokenizer fixes, same [ne] warm-start from [hi], just a different set of starting weights. Both are Chatterbox v3; only the pretrained T3 differs. A head-to-head is the cleanest way to settle which base transfers better to Nepali.

मेसिनले नेपाली बोल्यो 🗣 — Chatterbox v3 · Nepali LoRA fine-tune · Oshara AI engineering log. Every bug in this post was caught by listening.