The 60-case wall: nine attempts against a ceiling, and the one variable that brings it down.

In the first article of this campaign, a local 2.9-billion-parameter RNN reached parity with Qwen3-30B at tool calling for $0.53 of training. The open question was whether we could do better than 61 out of 82. Short answer: not with that model. We spent a week trying, six methods and nine measurements piled up between 58 and 61, and it took changing the one thing training cannot change to understand what was blocking.

Raw, a 7.2-billion-parameter RWKV-7 scores 29/82, roughly like the raw 2.9B (28). The same state-tuning from the first article, applied as-is for about $3, brings it to 69/82. That is eleven points above Qwen3-30B in its production configuration, and two points under the same Qwen when we lend it our retrieval scaffolding. As usual, everything is verifiable: raw generations, trained states, fingerprints, recipes, and this time the RL loop code as well.

Nine attempts, three points apart

After the first article we went at the ceiling with everything we had. Every attempt is in the public benchmark, generations included:

A few lessons came out of this series, and they are worth spelling out because they say what a small model can and cannot learn.

Targeted data did not do what we expected. We had built 135 examples aimed precisely at the seven recurring selection errors, validated against the schemas, filtered against the judge. None of the seven got fixed. The wrong answers moved around, model_get_info becoming model_manage, while abstention climbed to perfection without us aiming at it. An 11 MB state learns very well when to stay silent. It does not learn which of two similar names is the right one.

RL did not break through either, but it failed in an instructive way. We built a GRPO loop (group-normalized advantages, clipped surrogate, our mechanical scorer as the reward) on top of the champion, in two versions: keep training the state, or freeze it and put a fresh LoRA on top. Five million trainable parameters on one side, thirty on the other. Both converge to nearly the same profile, arguments 13, hard selection 16, simple around 14, for totals of 60 and 59. RL moved what data could not move, simple selection goes from 12 to 15, and paid for it in argument precision, at exactly the rate set by our reward, which paid selection twice what it paid arguments. The models maximized what we paid for, nothing else.

That is the main point of this series: the 58-61 band behaves like a frontier. Training chooses where you sit on it, never its level. You can watch it round by round in the LoRA version, arguments go 16, 17, then 13 while simple selection goes 12, 12, 14, and the total does not move. Early stopping does not help, the checkpoints taken at internal-reward peaks score 58 to 60 on the judge.

Two practical notes for whoever reuses the code: at LR 1e-4 the LoRA destroys the policy in one round, abstention drops from 8/8 to 1/8 (2e-5 is stable); and a 24-case probe does not predict an 82-case judge, never use it to select a checkpoint.

The variable that was left

When six methods plateau at the same place, the suspect is no longer the method. So we changed the size, and nothing else. The RWKV-7 G1g 7.2B checkpoint is from the same generation as our 2.9B, published three days before it. Same 2,920 examples, same state-tuning recipe, one epoch, same hyperparameters, same frozen judge. About $3 of L40S.

And so the result would mean something, the control demanded before publication: also evaluate the raw 7.2B, untrained.

So the wall really was the backbone. Six methods had not bought a single point above 61; the size change buys eight. And the categories that move are the ones the earlier series had pointed at: arguments go from 18 to 21, above the Qwen equipped with our top-40, and hard selection reaches 18/23, the best score of the campaign.

The control tells the other half of the story. Raw, the two checkpoints are twins, 28 and 29 out of 82, with the same signature, zero abstention and unparsable outputs. The raw 7.2B is no better at calling tools than the raw 2.9B. Size alone gives nothing on this task. It is the trained state that reaches what the larger model knows latently, and its effect grows with size: +33 points at 2.9B, +40 at 7.2B.

Which leaves the ranking everyone asks about. The 7.2-billion-parameter RNN beats Qwen3-30B in its production configuration, the one an operator would actually use, 69 to 58. With equal scaffolding it comes within two points of a model four times its size. The artifact carrying all of this is a 17 MB initial state, hot-loadable onto the public checkpoint.

The statistics, to close. The method effect is not up for debate: tuned versus raw at 7.2B, 42 discordant cases won against 2, p = 1.1×10⁻¹⁰ by exact McNemar. The size jump taken in isolation, 69 versus 61, gives 12 won against 4 and p = 0.077, at the edge of what an 82-case judge can settle on its own. What makes the finding solid is not that isolated test, it is the convergence: nine attempts locked inside three points, then one backbone change that exits them by eight. Both numbers are published, with the files to recompute them.

The architecture control, and it cuts both ways

That left the uncomfortable question: is any of this the RWKV architecture, or just our training? We took a size-matched transformer, Qwen2.5-7B-Instruct, and put it through the exact protocol of the previous rung. First raw on the judge. Then tuned on our 2,920 examples, same raw texts, LoRA standing in for state-tuning since a transformer has no state, two epochs, and judged again.

First verdict: raw, the transformer already knows the trade. 69/82 without any training at all, including a 20/23 at hard selection that no model in our campaign had reached, its 30-billion sibling included. The gap between the families is therefore not architecture, it is the pretraining corpus: Qwen2.5's is saturated with function calls, RWKV G1's has almost none. Our $3 state-tune closes exactly that gap, no more, no less.

Second verdict, the less expected one: transplanted as-is onto a model already tuned at the factory, our recipe does damage. 69 → 51, a clear degradation (23 cases lost against 5 won, p = 9×10⁻⁴), driven by 25 false refusals. The abstention curriculum the RWKV needed, a model that never stayed silent, teaches an already-calibrated model to stay silent on real requests. A Qwen-optimized SFT, with its chat template and its own hyperparameters, would probably not do this, and that is exactly the lesson: a training recipe is not a portable commodity, it fits a model. State-tuning fits RWKV. That fit is what +40 points for $3 buys.

What survives scale

Two weaknesses cross the size change. Simple selection stays average (13/18): the namespace collisions, linear_search_issues versus github_search_issues, survive scaling, which points at a format or retriever problem more than a capacity one. And multi-step stays at zero everywhere, no mystery there: our harness is one turn, one call, by construction.

The protocol limits are those of the first article, one benchmark, one task, our tools. The 7.2B run has not yet been replicated end-to-end the way rung 1 was, and the transformer control only got our naive recipe, not a carefully tailored SFT: both are open threads for whoever wants them.

And in production?

One last clarification, because a benchmark number and real usage are two different things. The 7.2B at 69/82 is not, today, our interactive production model: on an ordinary CPU, prefilling the 5,300 tokens of a top-40 shortlist through a 7.2B takes tens of seconds, which is why we picked the 2.9B at the start of this campaign in the first place. It takes on two roles instead: asynchronous agent work, where nobody is waiting at a screen, and the day a GPU enters the loop, the 17 MB state is ready.

The interactive production model remains the 2.9B at 61/82, and its next gain will not come from one more training run, the campaign has shown that well enough. It will come from state caching: an RNN can save its state after a fixed prefix and resume from there, and the mechanism measures in seconds per request. One engineering detail worth writing down: the GGUF format does not carry a trained initial state, serving goes through the native runtime. That is the kind of thing you learn by deploying, not by benchmarking, so it belongs here.

The rest of this story is the subject of a third article: how the architecture's creator showed us that our own protocol was biasing our measurements, and what the correction revealed.

Reproduce

Everything is in the benchmark repository (mirror on Forgejo): the 82-case judge, the schemas, the raw generations of every measurement in this article, including the ones that do not flatter us, the trained states with their SHA-256, the pod recipes, the GRPO loop, and the scripts that regenerate the figures from the committed JSON. The whole campaign, nine measurements, two RL runs, the 7.2B ascent and its control, cost about $25 of rented GPU. The detailed judge and anti-leakage protocol is in the first article.

Found a hole in the protocol? Write to us: contact@scarletwolf.ai. That is what publishing is for.