A full-duplex voice model listens and speaks at the same time — but out of the box it can only talk about what it already knows. It cannot check the date, search the web, or convert a currency. This is how we gave one a hand: grafting a tool-calling action stream onto PersonaPLEX, and the four training attempts it took to make it work.

Base: PersonaPLEX 7B (Moshi/Helium fork) · Method: LoRA + dense action head · Streams: 17 → 18 · Tools: 8

---

18	8	+5e-5	4
frame rows (was 17)	tool types	LoRA rank-32	training attempts

---

01 · What “tool calling” actually is

Text LLMs like ChatGPT don’t reach out to the internet by magic. They are trained to emit special tool-calling tokens — introduced first in Meta’s Toolformer — when a request needs one. The model already predicts a token at every step, so the trick is to add new tokens to the vocabulary that mean “a tool call starts here,” and teach the model to produce them.

A token like <|toolcall_begin|> signals that the tokens that follow describe a tool and its arguments; <|toolcall_end|> closes it. In between, the model writes something a runtime can parse:

<|toolcall_begin|>{"function": "get_weather", "arguments": {"city": "Kathmandu"}}<|toolcall_end|>

The runtime detects that call, executes get_weather deterministically, gets the result, injects it back into the model, and the model reads it out in natural language. The model never runs the tool itself — it only learns when and how to ask.

---

02 · Can a model that thinks in audio tokens do the same?

Yes — with one large caveat we’ll get to. PersonaPLEX (a fork of Kyutai’s Moshi) autoregressively predicts audio tokens, much like an LLM predicts text. Under the hood its temporal transformer, Helium, was originally a text language model — Kyutai trained it on text first, then adapted it to audio tokens produced by the Mimi codec.

So we can add tool-calling tokens to the vocabulary and fine-tune PersonaPLEX to use them, and the runtime can detect and execute them exactly as it would for a text LLM.

---

03 · The hard part: you can’t just pause a duplex model

Here is the caveat. A text LLM is turn-based — it can happily stop generating, wait for a tool to run, and resume. A duplex model cannot. It is producing an audio frame every 80 ms, continuously, in lockstep with the incoming user audio. Stepping out to run a tool, wait for a network call, and come back would tear a hole in that stream.

Moshi already solved a related problem — half-duplex to full-duplex — by modelling the user and the agent as two separate token streams, so both can “talk” at once. The natural extension for tool calling is the same move again: add one more stream. We call it the action stream, and the special tool-calling tokens live there, decoupled from the audio the user actually hears.

---

04 · The four changes that make PersonaPLEX a tool-caller

A PersonaPLEX input frame was a vector of 17 values: index 0 is the text inner monologue, indices 1–8 are the agent’s audio codebooks, and indices 9–16 are the user’s audio codebooks. Each stream has an input head that turns tokens into embeddings; the embeddings are summed into one vector fed to Helium. Adding an action stream means four changes to that picture.

---

05 · How we fine-tune it

Full fine-tuning a 7B duplex model to add one skill is overkill, so we use LoRA for the bulk of it and train only the genuinely new parts densely.

• Dense (full) fine-tune: the two new action layers, action_emb and action_linear — they start near-random, so they need real weight updates, on their own higher learning rate (action_lr = 1e-4).
• LoRA adapters (rank 32, α 64): Helium’s attention projections and gating FFNs, the depth-transformer’s in/out projections, and the text output head (on a gentler LR to protect the voice).
• Frozen: the Mimi codec, the audio embeddings, and the depth-transformer’s own attention — nothing that would risk the model’s existing voice quality.

Only ~4.2% of the 8.7B parameters are trainable. One more detail that matters at inference: the loss masks out the user’s audio codebooks — the model never generates those, so there is no reason to supervise them — and the training clips add noise to the user channel so the model learns to answer a real, noisy microphone rather than clean TTS.

---

06 · The dataset

The pipeline is the same one Kyutai ship in moshi-finetune, so the format matches theirs. Each training example is two files.

1. A stereo .wav — left channel is the agent, right channel is the user. Here is one real training clip (currency conversion). It plays the whole exchange:

2. A .json with the word-level transcription (alignments) and — this is the new part — an actions field. Each action carries the trigger time, the natural-language planning, the function and arguments the model must generate on the action stream, and the tool result (plus the result_time it arrives) that the model must speak. That result is the ground truth for the action-stream cross-entropy during training — the tool is never actually executed while training.

{
  "alignments": [
    ["What's", [0.0, 0.6222], "SPEAKER_USER"],
    ["75",     [0.6222, 0.8296], "SPEAKER_USER"],
    ["pounds",  [0.8296, 1.4519], "SPEAKER_USER"],
    ["worth",   [1.4519, 1.9704], "SPEAKER_USER"],
    ["in",      [1.9704, 2.1778], "SPEAKER_USER"],
    ["pesos?",  [2.1778, 2.8],    "SPEAKER_USER"],
    ["Let",     [3.3, 3.5662],    "SPEAKER_MAIN"],
    ["me",      [3.5662, 3.7437], "SPEAKER_MAIN"],
    ["...",     ["...snip..."],   "SPEAKER_MAIN"]
  ],
  "actions": [
    {
      "time": 3.3,
      "planning": "Let me convert that amount.",
      "function": "convert_currency",
      "arguments": { "amount": 75, "from_currency": "GBP", "to_currency": "MXN" },
      "result": "{\"converted\": 1731.82, \"to_currency\": \"MXN\"}",
      "result_time": 7.9911
    }
  ]
}

The interleaver reads that actions field and builds the 18th (action) row: idle <action_pad> on every frame, then at frame round(time × frame_rate) it writes the burst <|plan|> … <|toolcall_begin|> {json} <|toolcall_end|>, and at result_time it writes the <|tool_result_begin|> … <|tool_result_end|> result (whose positions are excluded from the loss — they’re observed input, not something the model should predict).

---

07 · Four attempts

We trained this in iterations. Each one fixed the previous failure and surfaced a new one.

---

08 · Inside the Attempt 3 run

The rest of this section is Attempt 3’s actual training run (~7,000 steps, 4 epochs, LoRA rank 32). Two things are worth watching: the loss, and what the model said at the eval probe every 100 steps.

The loss: the action head learns almost instantly

The action stream (orange) starts near ~11 nats — expected, since its output head is freshly initialised (§04) and has no idea what a tool-call token is. It collapses below 0.1 within ~140 steps and then rides the floor for the rest of training. The action-token grammar is the easy part; the model nails it fast. What’s left in the total loss is almost entirely the ordinary audio term (blue) — i.e. speech quality, not tool calling.

The eval probe: what the model actually said

Every 100 steps we ask the same spoken question and log the answer, the raw action-stream tokens, and whether the JSON parses. The eval question is:

Watching the action stream evolve tells the whole story of the run:

Step	Action stream	Parses?	What it means
100	no action emitted	—	Doesn’t call a tool at all — just hallucinates a name immediately (the Attempt-1 behaviour).
300	<|plan|> + {date""":""" Let"""time…	❌	It learned to fire, but the JSON is token soup.
900	{"arguments":{"query":"…"},"function":"web_search"}	✅	First valid call — correct function, correct shape.
3900–5000	valid web_search	✅	Consistently valid JSON. Format problem solved.
6800	{"arguments":{"query":"currency of Ukraine"},"function":"web_search"}	✅	Valid — but the query is wrong.

That last row is the punchline. By the end of the run the model emits flawless JSON and fires web_search every time — but for “Who is the Prime Minister of Nepal?” it searches “currency of Ukraine.” It has learned the shape of a tool call perfectly and copied a query it saw in training. Here is a real burst from step 6800:

<|plan|> I'll look this up online.<|toolcall_begin|>
{"arguments":{"query":"currency of Ukraine"},"function":"web_search"}<|tool_result_end|>
  -> VALID json, function='web_search'
  -> executed 'web_search' -> {"top_result":"Ukrainian hryvnia — ..."}  (injected)

09 · End-to-end demo — “What’s today’s date?”

The clearest single demo of the whole system is the one tool that needs no argument and can’t be faked: asking the model for today’s date, having it call get_current_date, and hearing it speak the real answer. Below is an actual offline run of the Attempt-3 checkpoint (adapter_epoch04_step6800.pt) — the same weights dissected above — driven by a real human-voice recording of the question (not the TTS training voice), left to drive the tool itself. Nothing is scripted; the date you hear is whatever the runtime’s clock returned at generation time.

Here is exactly what happened on the action stream during that clip, straight from the generation log:

[action]   <|plan|> emitted; get_current_date detected @frame 83
[toolcall] @frame 83  function='get_current_date'  arguments={}
[toolcall] executed 'get_current_date' -> {"date":"2026-07-17"}  (injecting tool result)
→ spoken: "Let's check, today's date. Today is July 17th, 2026."

The shape of it matches §08 precisely: a short plan/filler (“Let’s check, today’s date.”), the tool call fired on the action stream, a silent gap while the runtime executes get_current_date and injects {"date":"2026-07-17"}, and only then the spoken answer — with the real date, read out in the cloned voice. This is a tool that needs no arguments, so there’s no query to get wrong — which is why it works cleanly today, and why argument-bearing tools like web_search are what Attempt 4 is for.

---

10 · Where it landed

• A duplex model can call tools by adding a fourth stream — an action channel — with its own input embedding, output head, loss term, and vocabulary, and nothing about the audio path has to change.
• Timing is a first-class problem. A streaming model has no natural “wait” state; the runtime enforces one by holding text and feeding silence until the tool result is injected.
• Format is easy, content is hard. The action-token grammar and valid JSON were learned in a few hundred steps. Teaching the model to build the right query from the user’s question is the real work — and the reason for Attempt 4.

Next: finish the Attempt-4 run on the 4,000-example query-construction set, then repeat the end-to-end pass above on the tools that take arguments (web_search, convert_currency) — where getting the content right, not just the format, is the whole game.

---

PersonaPLEX · tool-calling action stream · Attempt 3 → 4 · © 2026 Oshara AI engineering log. The model learned to call tools perfectly — it just needs to learn what to ask.