The AI market sells itself on benchmarks. Almost none of those numbers can be checked by the person being shown them. Here are the four questions that make the difference, and our own results put through the same filter.
They apply to any vendor, us included. They require no technical skill, only a precise answer.
Was the test frozen before the measurement? A benchmark adjusted after seeing its own score measures nothing any more. The freeze date must precede the first number, and be verifiable.
Are the raw outputs published? Not the score table: what the model actually produced, case by case. It is the only way to recount for yourself, and to see the cases it got wrong.
Can a third party redo the calculation? The scorer, the data and the environment must be public and pinned to a version. Otherwise all you can do is believe.
Do they publish what failed? This is the hardest signal to fake. Anyone showing only successes either has outrageous luck or a drawer.
Frozen before measurement. Our judge holds 82 cases built on Gungnir's real tool schemas. The file has not moved since the first evaluation: no case was added or removed after seeing a score. The labels are mechanical, derived from the tool schema rather than from a model's judgment. And the anti-leakage is structural: 17 cases are hand-written, the other 65 were generated by a model from a different family than the one that produced the training data, then reviewed by hand, with automatic rejection of any query too close to the training set.
Raw outputs published. The generations of every run are in the benchmark repository, including the ones that do not flatter us. You can read what the model actually answered, case by case, and recount.
Reproducible, and not only in principle. The chain is deterministic end to end. We reran the full training and evaluation on a second machine: same scores, and the 164 generations identical character for character, with both trained states sharing the same SHA-256 fingerprint. It is therefore not an average with an error bar, it is an exact point anyone can recompute.
Failures published. Next section, because it deserves to be read in full.
The task: pick the right tool out of 231 and fill its arguments without inventing any, and call nothing when nothing should be called. That is the core job of a business assistant.
| System | Score out of 82 | Training cost |
|---|---|---|
| RWKV 2.9B, untrained | 28 | none |
| RWKV 2.9B state-tuned | 58 to 61 | $0.53 |
| RWKV 7.2B state-tuned | 69 | about $3 |
| Qwen3-30B cloud, production configuration | 58 | not applicable |
| Qwen3-30B cloud, with our scaffolding | 71 | not applicable |
An honest reading of the last two lines: our local 7.2-billion-parameter model beats the 30-billion cloud model in the configuration an operator actually uses, and stays two points behind once the cloud model is lent our own tool shortlist. Both numbers are published, because showing only the first would be exactly the trick this page is about.
The full detail is in the two papers: parity reached for 53 cents of training, then the ceiling, its map and the variable that brings it down. The retriever that makes the local pipeline possible has its own article too: a 144M embedder fine-tuned for zero euros.
The same protocol applies to work in progress. On Verdandi, where a model rewrites its own weights, measuring that identity survives first requires knowing what counts as noise: two answers from the same model to the same prompt can be nearly orthogonal, and a safeguard built on that would fire at random.
This section is the whole point of the page. It is published with the same level of detail as the successes, raw generations included.
None of these four results suits us. They are here because a result without its failures is an advertisement, not a result.
The benchmark repository holds the 82-case judge, the 231 tool schemas, the evaluation and scoring scripts, the raw generations of every run, the training curves, the machine-by-machine recipes, the RL loop code and the SHA-256 fingerprints of the models and trained states. The environment is pinned to a version.
Recounting an announced score requires neither a GPU nor an account: the generations and the scorer are enough, and it takes about ten minutes.
If you find an error in the method or in the numbers, write to us. That is exactly why all of it is published, and the correction will be public.
