XTTS v2 लाई भावना सिकाउँदा

We already had a Nepali XTTS v2 that read text cleanly. What it could not do was sound like it meant anything. This is the account of adding seven emotion tags to it — first in English, where a usable emotional-speech corpus exists, then in Nepali, where we had to build one out of films.

Two results are worth stating up front, because neither is what we expected. On English, LoRA and a full fine-tune came out nearly indistinguishable — LoRA marginally ahead on emotion, the full fine-tune marginally ahead on intelligibility — so a 173 MB adapter matched a 5.3 GB checkpoint, and we picked between them by ear. On Nepali, starting from a model that already knew English emotion bought us nothing measurable: there was no free transfer across languages, and the model learned Nepali emotion only from Nepali audio.

Base: Oshara Nepali XTTS v2 (epoch 10) · Method: <emotion> tag tokens · Data: EmoVoice-DB (en) + 1.83 h mined from Nepali film (ne) · Langs: en · ne

---

00 · How to read the samples in this post

One property of the training data shapes everything that follows, so it belongs before the results rather than buried in a caveat.

EmoVoice-DB pairs emotional wording with emotional delivery: its angry rows say angry things, its sad rows say sad things, and that is true for each of the seven emotions. So the tag and the text always agreed during training, and the model learned them together. It follows that the tags work best exactly where the training data lived — when the sentence itself is the angry one. Give it a flat sentence and ask for anger with the tag alone and you are asking for something it never saw.

Here is that difference, same model, same voice, same seed — only the wording changes:

We measured the gap over five seeds per tag. With emotion-matched wording the classifier agrees with the tag 29/35 of the time in English; with deliberately flat wording, 5/35. That is what the tags are for and where they hold, and every sample below uses emotion-appropriate text for exactly that reason.

---

01 · How you make a TTS model take direction

XTTS v2 has no emotion input. It takes text and six seconds of reference audio, and its prosody comes from whatever that reference happens to sound like. If you want angry, the conventional move is to hand it an angry reference clip — which means emotion control is really reference-clip curation, and you cannot ask for an emotion you do not already have a recording of.

We wanted emotion to be part of the text, so the API is a tag:

<angry> You promised me you would handle it, and you did nothing at all!

Mechanically this is small. Seven tokens are appended to the tokenizer’s vocabulary — one per emotion, taking ids 6681–6687 in a 6,689-token vocab — and the GPT learns what each one does to the audio tokens that follow. Nothing else about the architecture changes: the DVAE, the vocoder and the Perceiver resampler never learn that emotions exist.

The detail that decides whether any of this works

XTTS is conditioned on a reference clip, and by default the trainer cuts that reference out of the target utterance itself (gpt_use_masking_gt_prompt_approach=True). For emotion training that setting is fatal, and quietly so. If the prompt audio is a slice of the same angry sentence you are asking the model to produce, then the anger already reached the model through the audio, and the <angry> token has nothing left to explain. Training converges nicely and the tags control nothing.

So every row needs a neutral reference clip from the same speaker. The tag is then the only place the emotion can come from. Getting that right is trivial in EmoVoice-DB, which has neutral recordings of every voice. In Nepali film audio it turned out to be most of the work — section 07.

---

02 · A warning about the scorer, before any numbers

Every emotion number in this post comes from emotion2vec+ large (300M params, 42.5k hours) — the best open multilingual speech-emotion classifier we found, and the one our Nepali mining pipeline uses to label clips. It is also demonstrably a weak labeller, and we can put a number on that rather than hand-wave it.

While mining Nepali film audio (section 06) every clip got the classifier’s proposed emotion and a human reviewer’s verdict. Across 2,632 reviewed clips, the classifier’s label survived review unchanged 40.9% of the time:

Table view

proposed label	reviewed	kept	relabelled	rejected	kept %
sadness	215	135	17	63	62.8%
neutral	383	220	34	129	57.4%
anger	766	402	79	285	52.5%
happiness	1,105	305	192	608	27.6%
disgusted	56	8	20	28	14.3%
fearful	42	4	25	13	9.5%
surprised	65	2	27	36	3.1%

The failure is not uniform, and that is the part that matters here. On the four emotions that carry the dataset it holds up 43.0% of the time; on the three rare ones it manages 8.6% — surprised survived 2 times out of 65. It also over-produces neutral and has a pair of other/unknown heads it retreats to when unsure.

That is the honest frame for everything below. It is also the argument for the listening test we have not yet run.

---

03 · English: the same run twice, LoRA and full fine-tune

Starting from our Nepali XTTS v2 checkpoint (epoch 10 — the one from the earlier post), we trained the tags on EmoVoice-DB twice, changing only the adaptation method.

full fine-tuneLoRA

Table view

step	full FT	LoRA	LoRA ahead by
0	3.6914	3.6700	0.0214
500	3.0233	2.9555	0.0678
658	3.0109	2.9487	0.0622
1,000	2.9700	2.9155	0.0545
1,316	2.9574	2.9077	0.0497
1,500	2.9411	2.8922	0.0489
1,974	2.9299	2.8835	0.0464
2,000	2.9233	2.8726	0.0507
2,500	2.9146	2.8678	0.0468
2,632	2.9108	2.8661	0.0447
3,000	2.8998	2.8515	0.0483
3,290	2.8963	2.8532	0.0431

LoRA’s eval loss is lower at every checkpoint, and its artifact is 1/31st the size. On the metrics that matter downstream, though, the two runs split — and the margins are small enough that “split” is the honest word:

Metric	Full fine-tune	LoRA	Ahead
Emotion accuracy	0.412	0.427	LoRA, +0.015 (≈4 clips)
Macro-F1	0.356	0.378	LoRA, +0.022
Best eval loss	2.8963	2.8515	LoRA
WER (round-trip)	0.028	0.030	Full FT
CER (round-trip)	0.0059	0.0076	Full FT
ScoreQ MOS	3.799	3.709	Full FT
SQUIM MOS	4.182	4.215	LoRA
Clips scored / classifier declined	272 / 78	274 / 76	—
Trainable params	518.5M	122.6M	LoRA
Deployable artifact	5.3 GB	173 MB	LoRA, 31× smaller

Per emotion, LoRA’s whole advantage is concentrated in one place: surprised F1 0.280 against 0.130. Elsewhere they trade single clips — the full fine-tune is better on sad (0.421 vs 0.384), LoRA on happy (0.652 vs 0.619), and the rest is noise.

One number is worth pulling out: loss_text_ce finished at 0.0191 for both runs, identical to four decimals. Both methods learned the tag tokens equally well as symbols. The entire gap is loss_mel_ce — acoustic realisation. The tags were never the hard part; turning them into sound was.

---

04 · Before and after, in English

Seven emotions, seven sentences whose words carry them — the condition the tags were trained for. Each is rendered twice, with the reference clip, the seed and the temperature held fixed:

• Before — the base Nepali checkpoint. Its text_embedding has 6,681 rows: the seven <emotion> tokens do not exist in it, so there is no tag to give it and the untagged reading is the only “before” available. It is a competent narrator with no emotional control at all.
• After — the shipped full fine-tune, with the emotion’s tag on the front.

Scored against the tag that asked for them, the “before” column lands on the right emotion 2 of 7 times (neutral, and surprised — which the question mark in the text gets for free), and the “after” column 6 of 7. The one miss is fearful, which the classifier files as unknown rather than as a competing emotion — the same emotion, and the same classifier weakness, that section 02 warned about and section 05 measures.

---

05 · Scoring the English model

The shipped full fine-tune, on all 350 held-out EmoVoice-DB clips, scored with emotion2vec+ large. The classifier retreated to its other/unknown heads on 78 of them — those are refusals to name an emotion rather than competing guesses, so they are excluded and the numbers below cover the 272 clips it was willing to decide.

Emotion accuracy 0.412 · macro-F1 0.356, against 0.143 for a random pick among the seven labels.

Emotion	Precision	Recall	F1	n
happy	0.605	0.634	0.619	41
neutral	0.328	0.915	0.483	47
sad	0.471	0.381	0.421	42
disgusted	0.500	0.343	0.407	35
angry	0.478	0.324	0.386	34
surprised	0.250	0.088	0.130	34
fearful	0.200	0.026	0.045	39

Read the columns rather than the diagonal and the pattern is clear. 131 of 272 clips were classified neutral — under-acting is the default failure, and every emotion leaks into that column. happy is the strongest tag by a distance (F1 0.619); fearful and surprised are the two the model and the classifier fail on together.

Precision beats recall almost everywhere — disgusted 0.500 precision against 0.343 recall. When the model commits to an emotion it is usually right; the problem is that it often declines to commit.

Speech quality was never in doubt: WER 0.028, CER 0.006, ScoreQ 3.80, SQUIM 4.18 (round-trip through whisper-base.en). Whatever the tags did, they did not damage the voice.

---

06 · Nepali: there is no dataset, so we built one

For Nepali there is no EmoVoice-DB. There is no emotional-speech corpus at all. What there is, in quantity, is Nepali cinema — where people shout, weep and plead for a living.

So we built a mining pipeline. Point it at YouTube links or search phrases and it returns labelled, transcribed, single-speaker emotional clips:

Table view

stage	what happens here	clips	share of sliced
Utterances sliced	Silero VAD, 17 films, Demucs-separated vocals	13,928	100.0%
Survived the gates	content filters + per-emotion confidence gate	3,767	27.0%
Reviewed by a human	played back and judged in the review Space	2,632	18.9%
Accepted	label confirmed or corrected, kept as training data	1,470	10.6%

The yield is the point. 13,928 utterances in, 1,470 out at that snapshot — 10.6%. And the automatic gates were the aggressive part:

Rejection reason	Clips	Share
all_low — classifier not confident enough	6,991	68.8%
multi_speaker — more than one voice	1,629	16.0%
no_speech	986	9.7%
singing	449	4.4%
non_emotion / ambiguous / near_silence / wordless	106	1.0%

Review continued past that snapshot; the pool that actually trained the model was 2,070 accepted clips, 2,067 after dropping three longer than the trainer’s 11.5 s ceiling — 1.83 hours of Nepali emotional speech in total.

And it came out badly skewed

This is the single most important fact about the Nepali dataset, and it is a direct consequence of what Nepali films are full of. People shout and people joke; they are quietly heartbroken far less often, on camera, in a way a classifier will commit to.

This is the Nepali data that trained the model, per emotion:

Emotion	Train clips	Train audio	Share of train	Eval clips
angry	640	35.8 min	34.4%	33
happy	489	28.1 min	27.0%	30
neutral	451	22.1 min	21.2%	24
sad	355	16.8 min	16.1%	22
disgusted	13	0.9 min	0.8%	0
fearful	7	0.3 min	0.3%	0
surprised	3	0.1 min	0.1%	0
total	1,958	1.73 h		109

Angry and happy alone are 61% of it — 64 minutes between them. Sad gets under 17 minutes. And disgusted, fearful and surprised together come to 1.3 minutes across 23 clips, none of which even survived into the eval split: they are present in the tag vocabulary and absent from the data.

Remember this table when you read the Nepali results in section 09. It predicts them almost exactly.

The reviewer-agreement numbers from section 02 are the other half of this story. The three rare emotions collapse entirely — surprised survived review 3.1% of the time, fearful 9.5%, disgusted 14.3% — which is why the Nepali dataset ends up effectively four-class. It is not that Nepali films contain no fear; it is that we cannot yet find it at any useful precision.

---

07 · Who is speaking? Inventing the speaker labels

Now the problem from section 01 comes due. Every training row needs a neutral clip of the same voice as its reference, or the tag stops being load-bearing. But the mining pipeline records which film a clip came from, not who is talking — and a film is a whole cast. Hand-identifying a reference voice for every speaker across 17 films is not practical, and there is no cast metadata to lean on.

So we deduced the speakers from the audio, by clustering.

The run deduced 113 speakers across 17 films (6.6 per film), with emotion/cluster AMI at median 0.039, worst case 0.180 — so the clusters are about voices, with one film worth watching. No clip ended up alone in its cluster, which matters: a singleton has no same-voice partner to reference.

Turning clusters into references is a separate, deliberately conservative step. For each row, prefer a clip that is: from the same cluster, labelled neutral, not the row itself, never from the eval split (eval audio conditioning a training row is leakage of the kind that shows up as a good number), at least 6 s long (the trainer draws a random 3–6 s prompt slice, and a 2 s reference contributes one padded second), reviewer-confirmed over merely gate-passed, and least-used first — so one short clip does not become the prompt for two hundred rows and teach the model that its quirks are what the cluster sounds like.

What that yielded, on the 1,958 Nepali training rows that actually trained the model:

Reference for the row	Rows	Share
Neutral clip, same deduced voice ✅	1,637	83.6%
Neutral clip, different cluster in the same film	241	12.3%
The utterance itself	80	4.1%

196 distinct clips serve as references; the busiest serves 90 rows, the median 5. Per film, coverage runs from 100% down to 0% — three films had no usable neutral clip for any of their speakers at all.

---

08 · Does English emotion transfer to Nepali?

This was the experiment we most wanted to run. We had a model that had learned emotion in English. Emotion is often argued to be substantially language-independent — shouting is shouting. So: start from the English-emotion checkpoint and train on Nepali, and see whether the English knowledge gives a head start.

The setup:

• Base checkpoint: the shipped English-emotion full fine-tune — not the plain Nepali model. The tag tokens are already trained and already mean something.
• Training data: 1,958 Nepali rows (1.73 h) mixed with 839 EmoVoice-DB English rows — exactly 30.0%. The English is there as a rehearsal set: without it the English emotions drift away while the model learns Nepali ones.
• 11 epochs of 88 steps, batch 32 × grad-accum 2, lr 5e-6 with the same 1e-4 tag group, bf16.
• Nepali emotion distribution: angry 640, happy 489, neutral 451, sad 355, and then the cliff — disgusted 13, fearful 7, surprised 3.

Before and after, in Nepali

Here “before” is a real three-way comparison, not a single reading: the English-emotion model does have all seven tags trained — just never on Nepali. So we can ask it for <happy>, <sad> and <angry> on a Nepali sentence and hear exactly what English-only training transfers.

Both models get the identical Nepali sentence, the identical in-domain neutral film reference, and the same seed. The only difference is 528 steps of Nepali training.

The “before” column is the transfer result: the tags were fully trained, in English, and applying them to Nepali text does not deliver the emotion asked for. The English emotions did not carry across. The training curve says the same thing from the other direction — eval loss at step 0 was 4.0222, well above where either English run started (≈3.69), so the English-emotion model came to this data no better prepared than a fresh start would have been.

The “after” column is what 1.73 hours of mined Nepali film audio bought: delivery that reads as the emotion the tag asked for. That is the encouraging half of this experiment — the transfer did not happen, but the new data worked, and it worked on a language with no emotional-speech corpus at all.

How far did the Nepali data take it?

train loss (epoch end)eval lossgap

Table view

epoch	step	train	eval	train − eval
1	88	3.6337	3.5880	0.0457
2	176	3.4707	3.5463	-0.0756
3	264	3.4453	3.5246	-0.0793
4	352	3.4382	3.5115	-0.0733
5	440	3.3690	3.5008	-0.1318
6	528	3.3752	3.4738	-0.0986
7	616	3.3571	3.4932	-0.1361
8	704	3.3148	3.4888	-0.1740
9	792	3.2974	3.4895	-0.1921
10	880	3.2508	3.4864	-0.2356
11	968	3.2666	3.4854	-0.2188

Eval loss falls to 3.4738 at epoch 6 (step 528) and never beats it again, drifting to 3.4854 by epoch 11. Training loss keeps descending the whole way, 3.6337 → 3.2666. The gap between them goes from −0.046 at epoch 1 to +0.219 at epoch 11.

That shape has one honest reading: the model has extracted what 1.73 hours of Nepali audio has to give, and everything after epoch 6 is memorisation. Step 528 is the checkpoint we kept and the one scored below.

---

09 · Scoring the Nepali model

Same classifier, on the 109-row held-out Nepali split (108 scored; one generation came out degenerate). The split contains only four of the seven emotions — the pipeline could not mine disgusted, fearful or surprised in usable quantity — so those are not scored rather than scored as zeros. The classifier declined on none of the 108.

Emotion accuracy 0.454 · macro-F1 0.464.

Emotion	Precision	Recall	F1	n
neutral	0.531	0.708	0.607	24
angry	0.514	0.562	0.537	32
sad	0.583	0.318	0.412	22
happy	0.412	0.233	0.298	30

• angry and neutral genuinely work. 18/32 and 17/24 on the diagonal, both with precision above 0.51 — and they are the first and third best-resourced emotions in the pool. High-arousal and no-arousal are the two ends the model found first, and the two it had the most audio for.
• sad is precise but timid — precision 0.583, recall 0.318. When it produces sadness it is convincing; it often produces something flatter. This is the 17-minute problem. Sadness is the thinnest of the four emotions the model has any real data for, and it is audibly the weakest of the four: to our ears the angry and happy takes carry their emotion, and the sad ones frequently do not. We do not think this is a modelling failure so much as a missing 40 minutes of Nepali weeping.
• happy scores lowest (F1 0.298), confused toward angry nine times — which is the one place where the classifier and our ears disagree. Happy is the second best-resourced emotion and it sounds like it; what the numbers are catching is partly the classifier’s own happy/angry confusion, the same one that made a proposed happiness survive human review only 27.6% of the time in section 02, and that led reviewers to relabel 58 mined “happy” clips as anger. Loud-happy and loud-angry overlap in Nepali film comedy for the classifier as much as for the model.
• disgusted, fearful and surprised are not evaluated at all, because 1.3 minutes of training audio produced nothing to evaluate. Those three tags exist on the Nepali model and do not work.

Speech quality and the round-trip

Metric	English (full FT)	Nepali (step 528)
ScoreQ MOS (no-reference)	3.799	3.219
SQUIM MOS	4.182	4.054
Round-trip WER	0.028	0.344
Round-trip CER	0.006	0.137

The MOS drop is real but expected: the Nepali model was trained on Demucs-separated film audio, and separation artefacts are part of what it learned to reproduce. The English model learned from clean studio recordings.

The Nepali round-trip uses the fine-tuned Nepali IndicConformer from our own mining pipeline — the same model that transcribed the training clips — rather than Whisper. That choice is the difference between a number and noise: multilingual whisper-large-v3 returns WER 1.011 on this audio, i.e. more errors than reference words, which describes Whisper’s Nepali rather than our synthesis.

---

10 · What we actually learned

Emotion did not transfer across languages for free. Starting from a model that had thoroughly learned seven English emotion tags gave no measurable head start on Nepali: the zero-shot Nepali takes did not deliver the emotion asked for, and the step-0 eval loss was worse than either English run’s. Nepali emotion came from Nepali audio and nothing else.

But 1.73 hours of mined film audio was enough to get real emotion in Nepali. For a language with no emotional-speech corpus, built entirely from YouTube, angry and neutral are solid and sad is convincing when it commits. That is the result we would repeat.

And we can see the ceiling. The eval curve flattening at epoch 6 while training loss keeps falling is the cleanest signal in the project. It is not a hyperparameter problem to tune around; it is a data volume problem.

Per emotion, the quality tracks the minutes almost exactly. Angry got 35.8 minutes and works; happy got 28.1 and works; sad got 16.8 and is the one of the four that audibly does not carry its emotion; disgusted, fearful and surprised got 1.3 minutes between them and do not work at all. Nepali films supply anger and comedy generously and quiet grief sparingly, so the corpus we could mine is skewed the same way the model now is. The single highest-value thing we can do next is find more sad Nepali audio — not tune anything.

The data ceiling is a labelling ceiling, not an audio ceiling. There is no shortage of Nepali film. There is a shortage of clips we can confidently label — surprised survived human review 3.1% of the time. Three of seven emotions are currently unreachable in Nepali, which is why the Nepali model is a four-class model wearing a seven-class tag set.

The tags work where the data lived. Because every training row paired emotional words with emotional delivery, the tags are strongest on emotionally-worded text and weak on flat text (29/35 against 5/35 in English). If you want a model that can put anger into a neutral sentence, that has to be trained for deliberately.

The reference-clip problem is as important as the emotion data. 84% coverage of same-voice neutral references took ECAPA embeddings, emotion-nuisance projection and per-video spectral clustering. The remaining 16% is a known tax on the result.

Our evaluator is part of the experiment. A classifier whose labels survive human review 40.9% of the time is scoring every emotion number here, and on English it ranked the two adaptation methods the opposite way round from both our ears and the intelligibility metrics. Treat the numbers as a relative yardstick and the ears as the arbiter.

Where this goes next

The open question is the one we started with, asked better. Not “does English emotion transfer to Nepali for free” — it does not — but can a model be built so that it does?

1. Separate the tag from the language. <angry> is currently one token doing two jobs: naming an emotion, and carrying whatever acoustic realisation the training languages showed it. A conditioning path that factorises emotion from language-specific prosody is the obvious structural fix.
2. Train multilingually from the start, rather than sequentially. Sequential training is what produced the negative transfer result; the tag may only become language-independent if it is forced to be during training.
3. Mine for sadness specifically, and for the three missing emotions. The fix for Nepali sad is more Nepali sad, and general-purpose mining will not find it — the gate rejects quiet, low-arousal speech at exactly the rate that produced this skew. That means per-emotion thresholds tuned for recall on sadness, source material chosen for it, and a Nepali-tuned emotion classifier rather than a multilingual one.
4. Get a better evaluator. Everything in section 02 argues for a human listening test to calibrate against, and for not trusting a single classifier’s per-class numbers.
5. Add emotion-neutral text to the training mix, so the tag has to carry the emotion on its own at least some of the time.
6. Pick checkpoints on something emotional, not on mel cross-entropy alone.

---

Appendix · Reproducing the numbers

Both evaluations are two commands: synthesise the held-out split, then score it. --classifier emotion2vec-large is the seven-class multilingual scorer; --only-decided drops the clips it refuses to call an emotion instead of giving them a confusion-matrix column, and --only-present-tags keeps emotions absent from the split out of the report rather than showing them as all-zero rows.

# English — the shipped full fine-tune, on EmoVoice-DB's 350-row held-out split
python scripts/evaluation/evaluation.py \
  --eval-csv MyTTSDataset/metadata_eval.pre_bhavvani.csv \
  --model-config ./runs_xtts_nep_base/fullfinetune_emovoicedb/config.json \
  --out-dir eval-outputs-en-fullft

python scripts/evaluation/evaluate_metrics.py \
  --eval-csv MyTTSDataset/metadata_eval.pre_bhavvani.csv \
  --audio-dir eval-outputs-en-fullft \
  --classifier emotion2vec-large \
  --only-decided --only-present-tags --create-confusion-metric-image

# The LoRA run, for the section 03 comparison. Its adapter stacks on the frozen
# Nepali base, which config.py no longer points at, so name it explicitly.
python scripts/evaluation/evaluation.py \
  --eval-csv MyTTSDataset/metadata_eval.pre_bhavvani.csv \
  --model-config ./runs_xtts_nep_base/lora_emovoicedb/config.json \
  --base-checkpoint pretrained_model_xtts_nep/model.pth \
  --out-dir eval-outputs-en-lora

# Nepali — the step-528 checkpoint, on the 109-row YouTube-mined split.
# --language ne is required: that csv carries no language column, so it would
# otherwise fall back to config.LANGUAGE and synthesise Nepali text as English.
# --asr-backend indicconformer runs the round-trip through the pipeline's own
# Nepali recogniser instead of Whisper.
python scripts/evaluation/evaluation.py \
  --eval-csv MyTTSDataset/metadata_youtube_emotion_eval.csv \
  --model-config ./runs_xtts_nep_base/xtts_nepali_youtube_emotion-*/config.json \
  --language ne \
  --out-dir eval-outputs-ne-youtube

python scripts/evaluation/evaluate_metrics.py \
  --eval-csv MyTTSDataset/metadata_youtube_emotion_eval.csv \
  --audio-dir eval-outputs-ne-youtube \
  --classifier emotion2vec-large \
  --asr-backend indicconformer \
  --only-decided --only-present-tags --create-confusion-metric-image

Loss curves come from each run’s losses.json; the dataset funnel and reviewer agreement from yt_emotion_dataset/{rejected,dataset}.csv at the 2026-09-02 snapshot; the speaker figures from speaker_deduction/{clustering_report.json,speakers.csv} and the two run logs. The before/after grids and the wording comparison were rendered at temperature 0.75, repetition penalty 5.0, fixed seeds, one reference clip per language.

Sample sizes, in one place, because several of these are small. English emotion scoring 272 of 350 clips (78 undecided). Nepali 108 clips across 4 emotions (32/30/24/22) — one clip is 0.9 accuracy points. Wording comparison 35 takes per condition in English, 15 in Nepali. Human review 2,632 clips. Speaker clustering 2,605 clips across 17 films. Single training run per configuration, single seed, no error bars: treat small differences as noise.