A week ago we published that a recent generation of RWKV checkpoints was worse at tool calling than its predecessor. The number looked solid: 29 against 38 on our 82-case judge, dominated in every category, p = 0.012 on a paired test. We drew a general lesson from it about progress in generalist models, which would not be monotonic when you look at a narrow capability.
The conclusion was wrong, and our own protocol produced it.
This post is about how we found out, what the correction cost, and what it revealed afterwards. It is the third in a series: the first documented a $0.53 training run reaching parity with a cloud model, the second mapped that small model's ceiling and broke it by changing its size.
We had shared our raw numbers on the RWKV Discord, at the request of Peng Bo, the architecture's creator. 6 checkpoints, untrained, in our published protocol. His answer was three words: that's strange.
Then a suggestion. Our prompt template forces the model to produce JSON right at the assistant marker, and in his view that could penalise the newer models, trained on other formats. He proposed a way to check: let the model produce free completions from bare prefixes, and read back the function-call formats that come out. The model tells you itself what it was trained on.
So we did. 24 completions per checkpoint, temperature 1, no context. One checkpoint spontaneously rebuilds <tool_call> tags with conversation roles; another produces calls in a major API provider's shape, with a path for multiple calls. Neither looks like what we were imposing on them.
Those formats were in fact declared in plain code in an ecosystem repository we had not read. 3 checkpoints, 3 templates, and we were evaluating all three with the one that suited the oldest.
We re-evaluated each checkpoint in its own format, with the decode settings the ecosystem recommends.
The two newer checkpoints gain 6 and 9 points. The oldest, the one our format already suited, loses 3. That last number does all the work: without it there is no way to tell whether the gain comes from the format or from the new decode settings. It comes from the format.
The ranking changes as a result. The checkpoint we called a regression was level with its predecessor; the one from the following generation, which we called equivalent, is clearly ahead once asked properly. The progress existed, our protocol was hiding it.
We published the correction in both previous articles, with its date and mechanism, and announced it in the thread before anyone raised it. When the measurement and the conclusion disagree, the conclusion gives way.
It would be convenient to blame everything on the format. The data does not allow it.
On these raw checkpoints, whatever the template, abstention stays at 0/17. No untrained model has ever scored a point in that category, in any run, across 12 days of measurement. Multi-step stays at 0/2 as well. What the format fixed was syntax: unparsable outputs go from 16 to 1 on one checkpoint, from 26 to 2 on the other. Judgement does not move by a single point.
That is the line this campaign ended up drawing, and it separates two things we usually conflate.
The first article showed that $0.53 of training, touching 0.2% of the parameters, took abstention from 0/17 to 17/17 in a single epoch. The second showed that 135 examples built precisely against 7 tool-name confusions fixed none of them: the wrong answers moved to other wrong answers.
This week, two more measurements came from the same side.
We evaluated a distilled hybrid, a transformer converted to a recurrent architecture, whose teacher model happens to be our comparison model. The before and after were therefore exact, on the same judge.
The teacher knows perfectly well when to stay silent, 17/17. The distilled student drops to 4/17, and to 0/17 in the other configuration we tested. Asked something no tool can address, it calls a text-concatenation function and stuffs an explanation of network protocols into its arguments. Its content capability, on the other hand, travels rather well: 17 correct arguments out of 22, 14 simple selections out of 18. Respectable scores.
The hybrid's author had stated his prediction before the measurement: his distillation set contains few tool calls, everything is trained on a pure divergence objective, and in his view tool-call distillation requires training where the model produces its own responses. The measurement agrees with him and pins down where: what does not cross over is the reflex.
The second measurement comes from the other end. Making the model reason before the call degraded everything, abstention falling from 17/17 to 9/17 because it argues itself into acting. So we trained a state on 2,920 examples containing a reasoning block, 425 of which conclude explicitly that no tool applies. Aimed straight at the observed mechanism.
It recovers ground, from 58 to 61. It stays 10 points under simply asking directly, and abstention tops out at 11/17 despite the 425 reasoned refusals. On this task, making the model think before it decides costs more than it returns.
Four measurements, four different directions, the same result. Cheap training installs dispositions and does not fix fine-grained mappings. Distillation carries the mappings and loses the dispositions. Knowing when to do nothing is not transmitted like knowledge; it gets installed.
Along the way, two things we were not looking for.
Prompt length was costing us dearly. Our system block held the full JSON schemas of the 40 shortlisted tools, 34,000 characters before even reaching the user's question. Replaced by one line per tool, on a raw model, the score goes from 28 to 48. Twenty points with no retraining. But the easy conclusion would be misleading: removing descriptions entirely costs 9 points, mostly in abstention. It is not length as such, it is structure. The model needs to know what its tools do in order to decide not to call them.
And our training runs were wasting time. On the last one we evaluated the first-epoch checkpoint alongside the final model: identical score. The 2 following epochs added nothing. We had been training for 3 epochs out of habit, never having checked.
A benchmark measures two things at once: the model, and the way you question it. As long as you evaluate your own model, trained on your own format, the two blend harmlessly. The moment you evaluate a model you did not train, the gap between them becomes most of what you are measuring, and you cannot see it, because it looks like a result.
The safeguard is not complicated: you need a control. A model you know suits the protocol already, measured in both conditions. If it gains as much as the others, the protocol change explains everything. If it loses, as ours lost its 3 points, then the others' gain is real. Without that control, our correction would have been a second conclusion as fragile as the first.
The rest is hygiene: publish the raw generations so someone can recount, publish the numbers that hurt because those are the informative ones, and correct yourself before you get corrected.
Over these last three days this benchmark has served someone other than us. We measured checkpoints nobody had evaluated on this task, one of them published that same morning, at its author's request. We reported a corpus artifact, a roleplay tag grammar memorised word for word that surfaces when you scratch the model with no context, and the author reproduced it himself, pushing further than we had. A contributor is going to replay our judge on his own distilled models.
It is a modest role: measuring cleanly what others build. Building is more rewarding, measuring honestly less so, and that is probably why there is room left.
Everything is in the benchmark repository (mirror on Forgejo): the 82-case judge, the tool schemas, the raw generations of each of the 25 measurements in this series, the format probes, the trained states with their fingerprints, the pod recipes, and the scripts that regenerate the figures from the committed results. The whole campaign cost about $35 of rented GPU.
Found a hole in the protocol? Write to us: contact@scarletwolf.ai. The last one arrived in a three-word message.
