If you fine-tune a model several times in a row, one question always shows up: did I just break something? The reflex is to ask the same question before and after, read both answers, and draw a conclusion. We measured what that reflex is worth. It is worth nothing, and we can say by how much.
On our bench, two answers from the same model, to the same prompt, with two different seeds, can be nearly orthogonal: cosine similarity dropping to 0.079. An observer comparing those two outputs would conclude that a major drift had occurred. There is none: same model, same weights, same machine.
This post documents the protocol we built to separate signal from noise, as part of Verdandi, our research project on a model that rewrites its own weights. The problem is far broader than that case: it faces anyone who chains fine-tunes and wants to know what they lost along the way.
Verdandi explores an idea that is easy to state and unpleasant to validate: a model that consolidates its weights overnight, from what it experienced during the day. The thesis is public, the architecture stays in the workshop. But the validation question is universal.
If the weights move on their own, how do you know it is still the same entity in the morning? You need a test that fails when identity is lost, and that does not fire when nothing unusual happened. A test that cries wolf one night in three would not be a safeguard, only noise you would eventually ignore.
So we built a 250-item identity battery, split into three families:
Before measuring drift, you have to prove the measurement itself does not drift. Two complete greedy-decoding runs of the battery, 250 items each, produced outputs that were byte-identical, with zero divergence.
This is the prerequisite people skip too often. Without it, every observed gap is arguable: was it the model that changed, or the machine? With it, the two deterministic families become precision instruments, and any gap there is signal.
Their starting values: 95.00% on knowledge (190 out of 200) and 90.00% on normative (45 out of 50). That second figure includes three genuine compliance failures of the base model, which we left in the score. It is not a flattering starting point, it is the real one.
Style, however, requires sampled decoding. There is no way to freeze it without turning the model into a repetition machine. And that is where the naive measurement collapses.
Across 20 prompts drawn with 10 seeds, the noise from one sample to the next is massive. Mean sentence length varies by 38.7% from seed to seed. Punctuation density by 17.9%. And semantically, the mean cosine between two outputs of the same prompt falls to 0.688, with that 0.079 floor that makes direct comparison absurd.
A threshold set on single outputs would therefore have fired at random. That was the trap we were about to walk into.
The solution requires no extra compute, only looking at the same data differently. We split the 10 seeds into two independent halves of 5, then compared the aggregates of each half. That measures exactly what a safeguard would see between two nights with no real drift at all.
| Metric | Per-sample noise | After 5-vs-5 aggregation |
|---|---|---|
| Mean sentence length | 38.7% | 0.67% |
| Punctuation density | 17.9% | 1.17% |
| Word count | 14.0% | 2.46% |
| Character count | 12.6% | 1.75% |
| Lexical richness | 8.3% | 0.52% |
Semantically, the same move changes everything: the cosine between the centroids of the two halves is 0.912 on average, with a floor at 0.825. The floor went from 0.079 to 0.825.
Five seeds are therefore enough. That is the result that makes the rest possible: you can place a threshold well above 2.5% of noise and know that a trigger means something.
One detail is worth noting for anyone redoing the exercise: the least stable prompts are open-ended French genres, explanation, opinion, reflection, with a floor at 0.825. The most stable are English description prompts, up to 0.969. A drift showing up only on open French prompts would therefore be suspicious before being alarming.
Measuring that identity is preserved is not enough. A model with a learning rate of zero would pass that test perfectly, every night, indefinitely. It would have learned nothing, forgotten nothing, and the safeguard would show green.
The test must therefore be dual: identity preserved and acquisition demonstrated. Both halves, or neither.
That is where we stand, and it is the honest limit of this post: the identity half is calibrated, the acquisition half is not. It requires a corpus of night episodes that does not exist yet. Until it does, no cycle has run, and we claim nothing about what a self-evolving model would produce. What we are publishing here is the instrument, not the result.
A full battery pass takes about 2 h 20 on an A40: 13 minutes for the two deterministic families run twice, and 124 minutes for style. The reason is that greedy decoding runs at 49 tokens per second where sampling tops out around 10.
We thought we had the explanation: the full vocabulary sort at every token. A version with a bounded sort was written, measured, then rejected on two counts. The gain was 1.15×, negligible. And above all the outputs were no longer identical, which would have invalidated the baseline everything else uses as its reference.
The real bottleneck was elsewhere: at every token, sampling pulls 65,536 floats back from the GPU to the CPU, where greedy decoding does its work on the GPU. The hypothesis was wrong twice over. We reverted to the implementation that produced the baseline, with a warning comment so nobody attempts the same cleanup again.
It is a negative result with no commercial value, and it is here for the same reason as the others: a result without its failures is an advertisement, not a result.
The battery, the measurement harness and the raw outputs live in the Verdandi repository, which is not public yet: publication will follow the end of the current phase, once the battery is frozen. In the meantime we gladly share the protocol and the variance report on request, write to us.
The rest of our work is public and verifiable right now, starting with the tool-calling benchmark and its follow-up. Our measurement method, and the list of what failed, are on the Evidence page.
Verdandi is a research project of Scarlet Wolf, distinct from the Gungnir assistant which does run in production. The thesis is public, the architecture stays in the workshop.
