Can you teach a full-duplex speech model brand-new knowledge — and have it speak that knowledge back, out loud, in a cloned voice? This is how we did it: turning a Shakespeare play into spoken question-answer data and LoRA-fine-tuning PersonaPLEX in document mode until it recites grounded lines and attributions on request.

Base: PersonaPLEX 7B (Moshi/Helium fork) · Mode: document · Method: LoRA r=64 · Data: 1,500 stereo clips

---

1,500	~17	r=64	2.46%
training clips	epochs	LoRA rank	of params trained

---

01 · The goal: new knowledge, spoken

PersonaPLEX is a Moshi-style, full-duplex speech-to-speech model — it listens and talks at the same time, modelling several parallel token streams per 12.5 Hz frame. It’s fluent and conversational out of the box, but it doesn’t know the text of Coriolanus. We wanted to inject a specific corpus of knowledge and have the model both understand a spoken question about it and answer with the right lines, as speech.

That’s what document mode is for: continued fine-tuning on conversational speech, where each training example is a short spoken exchange (user asks → agent answers). No new streams, no architecture surgery — just teaching the existing model new material through its normal text-and-audio channels.

---

02 · What we fine-tune

The base model, nvidia/personaplex-7b-v1, runs a 17-row frame in document mode:

Row	Stream	Notes
0	text (inner monologue)	agent (SPEAKER_MAIN) words only
1–8	agent audio codebooks	Mimi codes, left channel
9–16	user audio codebooks	Mimi codes, right channel

Confirmed at launch: codebooks=17 dep_q=16 action=False, 8.58 B parameters total.

We don’t touch most of them. We attach LoRA adapters and train only those — ≈ 211.5 M / 8.58 B = 2.46 % of the weights — mirroring the kyutai-labs/moshi-finetune recipe adapted to the NVIDIA fork.

---

03 · The data pipeline

The whole game is producing spoken QA that is grounded — answers that are verbatim source lines or true attributions, so the training targets can never hallucinate.

---

04 · Dataset structure

Each example is two files, the standard moshi-finetune layout. Here is one real clip — a recite example. The stereo wav has the agent on the left, the user on the right:

The sibling .json holds word-level alignments — [word, [start, end], speaker] — with agent words tagged SPEAKER_MAIN and user words SPEAKER_USER:

{
  "alignments": [
    ["Recite",  [0.0, 0.5467],   "SPEAKER_USER"],
    ["exactly", [0.5467, 1.1844], "SPEAKER_USER"],
    ["the",     [1.1844, 1.4578], "SPEAKER_USER"],
    ["lines",   [1.4578, 1.9133], "SPEAKER_USER"],
    ["SICINIUS",[1.9133, 2.6422], "SPEAKER_USER"],
    ["...",     ["...snip..."],   "SPEAKER_USER"],
    ["Nature",  [5.90, 6.28],     "SPEAKER_MAIN"],
    ["teaches", [6.28, 6.79],     "SPEAKER_MAIN"],
    ["...",     ["...snip..."],   "SPEAKER_MAIN"]
  ]
}

The interleaver turns this into the [1, 17, T] codes tensor: the text row is built from the agent’s word timestamps (keep_main_only — user words stay audio-only, so the text stream is ~85 % padding, which is normal at 12.5 Hz), and the 16 audio rows are the Mimi encode of the two channels.

---

05 · Training configuration

Setting	Value	Why
mode	document	text + stereo audio, no action stream
lora_r / lora_alpha	64 / 64	scaling 1.0 (conservative)
lora_lr	2e-5	scaled down for multi-epoch knowledge learning
text_linear_lr	2e-5	separate group protecting the text head
duration_sec	10	clip window
warmup	350	pct_start ≈ 0.01
batch	1	moshi-finetune default
user_noise_prob	0.8	real-mic robustness
mask_user_audio	true	agent-only audio loss
text_padding_weight	0.5	content-vs-timing balance
weight_decay / grad_clip	0.1 / 1.0	matched defaults

Launch:

PYTHONPATH=$PWD/moshi:$PWD python -m training.train --config configs/document.yaml

---

06 · The loss curves

The run reached ~17 epochs (~35.6k steps) over the 1,500 clips.

The two streams tell different stories. The text (inner-monologue) loss falls below ~0.05 within a few epochs and stays there — the model learns to write the answer fast. The audio term is the slow, remaining signal: it drifts down more gradually and is what most of the total loss reflects by the end. (The bump after ~29k is the OneCycle LR being re-warmed when the run was resumed and extended — not a regression in the model.)

---

07 · Listen: the model answering

These are actual offline runs of the latest checkpoint (adapter_epoch17_step35500.pt), each fed a spoken question and left to answer. All three come back word-for-word correct against the source — a recite, an attribution, and a verbatim quote:

Each clip mixes the spoken question and the generated spoken answer on one track: the user asks, and after a short beat the agent answers in the cloned voice — with the correct line.

---

08 · Where it landed

• New knowledge, spoken. A 2.46 %-of-parameters LoRA in document mode taught PersonaPLEX to recite grounded Shakespeare lines and attributions aloud, on demand, from a spoken question.
• Grounded data can’t teach hallucination. Because every training target is a verbatim source line or true attribution, the supervision is always correct — the model’s job is to recall and speak it, not invent it.
• Text is learned fast; audio is the long tail. The inner-monologue loss bottoms out in a few epochs; the remaining signal — and remaining polish — is in the audio stream.
• Built for real mics, not just TTS. User-channel noise augmentation plus masking the user audio from the loss are what keep it answering noisy, real-microphone input instead of going silent out of distribution.

Not every checkpoint or every question lands perfectly — longer recitations can still drift after a correct opening — but the recipe reliably produces grounded, spoken recall. The open item is finishing the LR anneal for a final polish and a broader listening pass across instruction types.

---

PersonaPLEX · document-mode LoRA · Shakespeare QA · © 2026 Oshara AI engineering log. New knowledge, learned quietly in the text stream and spoken out loud.