Oshara · Engineering
Notes from the voice-AI trenches
From-the-trenches notes on building voice AI at Oshara — the bugs, the fixes, and what we learned shipping them.
Teaching XTTS v2 to Feel: English First, Then Nepali
We gave XTTS v2 seven emotion tags and trained them on English, where a good dataset exists. Then we tried the same thing in Nepali, where none does — mining our own corpus out of Nepali films, clustering the cast to invent the speaker labels the data never had, and finding out that English emotion does not transfer for free.
Fine-Tuning IndicConformer for Nepali Speech Recognition
We benchmarked AI4Bharat's IndicConformer against six other Nepali STT systems, then fine-tuned it further — hitting a tokenizer bug along the way that taught us something useful about training data hygiene.
Two LLMs Are Better Than One: How We Build Low-Latency Voice Agents
The awkward pause in voice AI is a timing problem, not an intelligence problem. This is the two-parallel-LLM architecture we use to fix it — a fast model that buys presence, a smart model that carries the answer — plus a tour of how a real-time voice agent is actually assembled.
A Nepali Voice for XTTS v2
Nepali has no open, zero-shot text-to-speech system. We built one by fine-tuning XTTS v2 — reusing its existing Hindi tokenizer instead of adding a single new vocabulary token — and it's now the best open, self-hostable Nepali voice we could measure.
Building an Agent, Step by Step
A complete walkthrough of creating a voice agent on Oshara.ai — from the My Agents page all the way to a deployed, embeddable widget.
Teaching a Duplex Voice Model to Call Tools
PersonaPLEX speaks and listens at the same time, but it can't look anything up. This is how we grafted a tool-calling action stream onto a full-duplex audio model — the architecture changes, the four training attempts, and why it kept searching for the wrong thing.
Teaching PersonaPLEX to Recite Shakespeare
How we injected new knowledge into a full-duplex speech model with LoRA — turning a Shakespeare play into spoken question-answer data, and fine-tuning PersonaPLEX in document mode to recite grounded lines and attributions out loud.
Teaching XTTS v2 to Speak Nepali
How we fine-tuned XTTS v2, a zero-shot voice-cloning TTS model, into a fluent Nepali narrator: the architecture, why epoch 10 beat epoch 30, how it stacks up against a different TTS model, and the text-pipeline fixes that stopped it from babbling.
Teaching a Machine to Speak Nepali
A from-the-trenches account of fine-tuning Chatterbox v3, a voice-cloning TTS model, for Nepali — the tokenizer bugs, the phantom sentences, the overfitting hunt, and the audio that came out the other side.