Can a 144M RNN compete with the best transformer embedders?.

TL;DR

• Our retrieval benchmark (56 French queries over English technical docs) had crowned granite-embedding-107m (53/56 hit@3) and buried the RWKV embedder at 26/56.
• False lead: that 26 was a truncation artifact. At equal context, the RNN actually starts at 48/56.
• Three nights of targeted fine-tuning later (free GPU, synthetic training queries, €0): hit@1 43 versus 39 for granite, best score of the panel, including models 2 to 4 times bigger. A tie on the production metric, and a slightly better anti-hallucination gate (0.938 vs 0.932).
• On a never-seen external judge (MIRACL-fr), the adaptation broke nothing: 95.6% versus 95.9% hit@1 for the original model.
• Then the harshest judge spoke: at full scale (46,000 chunks, exact production config), the advantage flips. The switch did not happen. Pooled benchmarks were overestimating the RNN, and we measured why.
• While preparing the counter-attack, we flushed out a bug that collapsed every long text onto a single vector (a measured cosine of 1.0000 between long Python code and a pie recipe): the EOS token did not survive truncation. Fixed, published, reported upstream. And ~20% of our training data had been mute from the start.
• The final round (hard negatives mined on the real corpus, corrected pipeline) did exactly what we asked of it: +4 hit@1 over its predecessor, fewer false refusals. But the underlying gap with granite cannot be closed with light fine-tuning: granite stays in production.
• The RNN still leaves with a crown: out-of-corpus refusal, 22/24 versus 17/24. On "knows how to say I don't know", Morrigan's founding bet, the small model fine-tuned for €0 beats the reference transformer.
• Out of five training runs, two failed outright, one missed its target, and the final verdict proved us wrong about our own favorite, twice. All of it is documented here, because that is where we learned the most, and it is precisely this protocol that protected production.

---

1. Context: why the embedder is the vital organ of strict RAG

Morrigan, our sovereign AI assistant, rests on one bet: zero structural hallucination. Without documentary context judged reliable, the generation model is never called, and the system answers "I don't know". That bet puts the entire weight on retrieval. A mediocre embedder does not degrade answers: it suppresses them (wrongful refusals) or poisons them (bad context presented as reliable).

The use case is demanding: questions in French over technical documentation in English (man pages, MDN, Python docs). Cross-lingual, jargon-heavy, federated corpora.

And we had a thesis constraint: Morrigan generates with RWKV-7, an RNN, not a transformer. Linear cost, constant state, 100% CPU. The uncomfortable question: could retrieval also run on this architecture without sacrificing quality?

2. The shootout: establishing who is best (it was not us)

Method: a fair 5,000-chunk subset of our production corpus (46,569 in total), relevant passages guaranteed present, the same 56 French queries for every model, comparable hit@k. The eval set is versioned and has never been used for training anything.

¹ chunks per second on an i5-10210U (laptop); absolute values depend on throttling, relative ones are reliable.

An unambiguous verdict: granite-107m offers the best precision/weight/speed ratio in its class, only 2 hits behind the 278m, which costs 2.6 times its weight and remains unusable on CPU. It went into production. The leader of the 100-150M class is granite.

And the RWKV embedder? 26/56 for the 0.1B (measured with the recommended query instruction, our engine's default; 18/56 without), 22-27 hits below the top of the table. The original paper (EmbeddingRWKV, arXiv:2601.07861) nevertheless reported an English MTEB of 63.06, within 0.8 points of a same-size transformer (gte-base: 63.90). But multilingual was not evaluated there: our bench was, to our knowledge, the first cross-lingual measurement of this model. Provisional conclusion: discarded.

3. The twist: the 26/56 was a measurement artifact

A footnote in our own bench doc read: "RWKV evaluated at a 256-token cap (CPU time concession) vs 512 for the others; if RWKV is close, a fair-512 re-run is recommended". At 26/56, nobody believed in it. We did it anyway.

² gate = balanced accuracy separating in-corpus / out-of-corpus; our measure of "knows how to say I don't know".

+22 hits by lifting the cap. The explanation is architectural. A mean-pooling transformer averages all tokens: truncating the passage costs it little. A causal RNN pools its final state, built left to right: truncating mid-passage amputates half the content from the embedding. An evaluation lesson that goes beyond our case: a "reasonable" truncation cap can crush a recurrent architecture by 22 points without any error being visible anywhere.

On equal footing, the 144M RNN was in fact tied with e5-small and embeddinggemma-300m, beat e5-base (278M), and stood 5 hits from granite. The campaign became winnable.

4. The campaign: five runs, one hypothesis per run, €0

Rules set in advance: a single hypothesis per run (never two changes at once), the local bench as sole judge, every checkpoint kept, failures documented. Compute: Kaggle's free tier (T4, ~30 h/week) for training, the laptop for inference and verdicts. Light fine-tuning: retrieval head plus the last 2 blocks (16.3M parameters trained out of 144M).

Round 1, the instructive failure (FR→FR data)

60,000 pairs from mMARCO's "french" config plus 20,000 CodeSearchNet, in-batch InfoNCE. Result: 47/56 (best), 43/56 (final), a regression below the 48 baseline that worsens with training time (48, then 47, then 43: a clean dose-response curve).

The diagnosis fit in one line: mMARCO's "french" config provides FR→FR pairs. Our bench measures FR→EN. We trained the model to be monolingual, and it dutifully unlearned cross-lingual retrieval. Second flaw: 8 pairs per batch, so 7 negatives per positive; the loss collapsed to ~0.005 within 300 steps, no signal left.

Round 2, the causal fix (real FR→EN pairs)

Since mMARCO's translations are aligned by identifiers, zipping the "french" and "english" streams yields exact (FR query, EN passage) pairs. 50,000 FR→EN pairs, 10,000 FR→FR for retention, 10,000 CodeSearchNet, batch of 24 (23 negatives). One epoch.

Result: hit@1 39, parity with granite, hit@3 48, gate 0.920.

Round 2b, the other instructive failure (one epoch too many)

Same data, second epoch. The internal validation score was still climbing (0.902 to 0.910), and the bench went backwards: 36 hit@1, 47 hit@3. Over-specialization to mMARCO's style, drift from the target. Campaign law #4: at these volumes, one epoch, no more. A rising validation proxy proves nothing about the targeted skill.

The autopsy: knowing exactly what to attack

Before the final round, we dissected the 8 remaining hit@3 failures and cross-referenced them with granite's failures on the same queries:

• 2 failures shared with granite ("regular expressions in Python", "fetching remote changes with git"): hard for both architectures, probably the bench itself.
• 6 failures that are ours alone, which granite gets right, therefore proven winnable. All of the same type: unmapped FR→EN technical vocabulary ("fonctions fléchées" to arrow functions, "réseaux Docker" to network).
• Bonus: granite fails on "using dictionaries in Python", which our model gets right. The two architectures do not fall in the same places.

Round 3, the synthetic query factory

Domain vocabulary cannot be learned from generalist web pairs. So we generated our own data: 1,052 synthetic French queries produced by an open LLM we run ourselves (Qwen2.5-7B, on the free GPU) from the chunks of our own corpus. Around 600 chunks targeted (400 aimed at the domains of the 6 failures, 200 drawn at random), of which 547 produced retained queries, with a strict anti-leak guard: any generated query too close to a query from the eval set is discarded (Jaccard overlap above 0.5 = trash), zero leakage verified.

Final recipe: domain queries ×3, plus 8,000 FR→EN retention pairs, starting from the round 2 champion, a single pass, gentle learning rate.

5. The results

On our bench (the production domain)

³ hit@2 = the production metric: Morrigan only injects the top 2 chunks into the generation prompt.

Reading it:

• hit@1: 43. Best score of the panel, all sizes included, ahead of granite-278m (41) and the 560M e5-large (41). On "the right document in first position", the 144M RNN beats transformers 4 times its size.
• hit@2: 47 across the board. Perfect tie with granite on what the generator actually sees.
• gate: 0.938 versus 0.932. The "zero hallucination" mechanism is slightly better with the RNN, and 56 out of 56 legitimate queries pass the threshold.
• hit@3: granite keeps +2 (53 versus 51). It is the only column where it still leads, and we say it as loudly as the rest.
• The RNN's full trajectory over the campaign: 26 then 48 then 51 in hit@3, 22 then 43 in hit@1.

On an external judge (MIRACL-fr, never seen in training)

343 queries from the French MIRACL dev set, a pool of 5,000 Wikipedia passages (731 gold passages guaranteed present, distractors sampled uniformly from the corpus's 14.6M). Generalist domain, monolingual French: everything our fine-tune did not target.

Two answers in this table:

1. The domain adaptation broke nothing: minus 0.3 points of hit@1 compared to the original model, i.e. one query out of 343. The FR→EN retention slice in the training mix did its job.
2. On generalist ground, granite keeps ~2 points. The RNN is competitive away from home and dominant at home. That is exactly the profile you expect from a successful adaptation, and the trade-off is measured, not assumed.

At that point, everything said: switch. Two judges out of two, parity in production, retention intact. That is when we demanded a third judge. It changed everything.

6. One judge too many: full scale reverses the verdict

All the previous numbers came from pooled evals: 5,000 passages, relevant ones guaranteed, sampled distractors. Before touching production, we imposed a switching gate: the same duel, but on the full stacks: 46,569 code chunks plus the three system corpora, federated retrieval, production gates, zero sampling.

The subset advantage inverted. Everyone drops when going from 5,000 to 46,000 candidates, but not at the same speed: granite loses 2 to 4 hits, the RNN loses 7 to 8. And it wrongly refuses almost twice as often (9 versus 5).

The mechanism, once seen, is limpid. The fine-tune learned attraction (pulling the query toward the right document), and that holds at any scale. It never learned separation: our contrastive batches pitted each pair against 23 random negatives, never against the near-neighbors that swarm in a real corpus (the 40 MDN pages about functions, the 15 nearly identical network man pages). In a pool of 5,000, few lookalikes: attraction is enough. In 46,000, the lookalikes decide the ranking. And there, granite's "sharpened" space, forged by IBM on massive volumes with hard negatives, makes the difference. An aggravating factor: the RNN's cosine distribution is compressed (~0.2-0.6 versus ~0.3-0.9), less dynamic range to settle near-ties.

Decision: no switch. Our rule, written before the campaign, said: the thesis wins ties and small defeats, not clear ones. This one is clear. granite stays in production. The protocol did exactly what it exists for: preventing us from switching on beautiful pool numbers.

This result is also, to our knowledge, a new data point: pooled evaluations systematically overestimate an RNN embedder relative to a transformer. The gap in scale sensitivity only appears in real conditions. If you benchmark different architectures on reduced pools, this bias concerns you.

One card remained, the right one: a round 4 with hard negatives mined at full scale, training the model against its own lookalikes, exactly the technique that forged the reference embedders. But while preparing that round, we first found something else.

7. The interlude: the bug that collapsed every long text onto a single point

An outside hint tipped us off: how do we handle the end-of-sequence token (EOS) at truncation? Once checked, our preprocessing (CPU port and training lib) encoded with EOS then truncated at the cap. Any text longer than the cap therefore lost its EOS at the cut.

Yet EmbeddingRWKV's pooling reads the hidden state only at EOS positions. EOS absent → empty pooling mask → null vector → the head projects the same constant embedding for every text that is too long. No error, no NaN, no warning. Measurement: a cosine of 1.0000 between a long Python code text and a pie recipe. After the fix (truncate to max_tokens − 1 then append EOS): 0.159.

The magnitude, measured on samples of 400 chunks per corpus:

In other words: one training passage in five from rounds 1 to 3 carried no signal, or worse, pulled the query toward the constant point. The fair-512 of §3 is partly explained by this very mechanism: raising the cap from 256 to 512 pulled every 257-512-token chunk out of the constant point. The round 4 mined data was, for its part, clean (0 negatives out of 6,292 beyond the indexing cap, verified).

The fix is two lines; it is published in the CPU port and reported on the model's discussion thread. The lesson goes far beyond our case: an embedder with terminal-state pooling does not degrade gracefully under truncation: it collapses silently. A mean-pooling transformer loses information; an RNN that reads its state at EOS loses everything, at once, without saying so. If you truncate, check that your readout token survives the cut.

8. Round 4: hard negatives facing the harshest judge

Round 4 attacks the §6 mechanism: each query trains against its own lookalikes, the 6 documents that actually trapped it, mined from the champion's top-k errors on the full stack. In numbers: 1,052 domain queries × 6 hard negatives (6,292 in total), each pair seen 3 times with different lookalike draws, plus 8,000 FR→EN retention pairs (7,744 for training, 256 for pure validation). InfoNCE loss with explicit negatives: batches of 12 pairs × (1 positive + 3 sampled hard negatives) = 60 sequences per step, ~908 steps, a single pass, LR 1e-5, still 16.3M trained parameters, with the corrected preprocessing from §7.

A note on method, in all fairness: our original full-scale judge was lost in the meantime (a script left in a temporary directory, one reboot; a judge is code: version it). It was rebuilt stricter: hits counted on raw federated search, the refusal verdict rendered by the real production gate (cosine threshold + cross-language guard + lexical guard), each model's threshold calibrated by balanced accuracy on its own scores. The three contenders were re-judged under strictly identical conditions: full production stacks, 49,795 chunks (46,569 code + 1,988 sysadmin + 626 security + 612 network). The numbers that follow are therefore not comparable line by line with those of §6. Among themselves, they are perfectly comparable.

⁴ hit@2 = the production metric (2 chunks injected into the generation prompt).

Three readings, in order:

1. Hard negatives work. With an identical judge, round 4 gains +4 in hit@1 over round 3 and cuts false refusals from 7 to 5. The autopsy confirms the mechanism: the remaining failures concentrate on the zones the 1,052 domain queries barely covered (man pages, SQL; generation mostly targeted MDN, 544 queries versus 49 for man pages). The technique works; coverage is what was missing.
2. The gap with granite is structural. −11 in hit@1, −9 on the production metric. At near-equal size (144M versus 107M), it is IBM's massive multilingual training that speaks, not size. No realistic wave of fine-tuning with 16M trained parameters closes that. The remaining levers (full backbone, distillation, the never-properly-evaluated 0.4B) are research bets, no longer a race to the switch.
3. The RNN leaves with a crown: refusal. 22 out-of-corpus queries out of 24 refused, versus 17 for granite, which lets almost one off-topic question in three through to the generator. On "knows how to say I don't know", Morrigan's founding bet, the small RNN fine-tuned for €0 clearly beats the reference transformer. The score distributions tell the same story: after hard negatives, out-of-corpus peaks at 0.62 while relevant content climbs to 0.83: the model learned to widen the gap.

Final verdict: granite stays in production. The rule written before the campaign ruled twice in the same direction, and that is exactly as it should be: a protocol that can only say yes is not a protocol. The campaign closes here. The program continues, at a pace of one hypothesis per month: the never-trained full backbone, the never-attempted distillation, the 0.4B never tested under clean conditions, and a generalist multilingual embedder for the community, with exactly this recipe.

9. What it costs, what it changes

• Total campaign compute: €0. About 22 h of T4 on Kaggle's free tier (6 training launches, including one invalidated by the EOS bug and relaunched, plus 1 for generation), the rest on an i5 laptop with 16 GB of RAM.
• Inference speed: 2.3 to 3.4 chunks/s on laptop CPU at 512 tokens depending on thermal state (granite: ~2.7 to 5.3). Re-indexing the 49,795 chunks of the four corpora takes ~5 h 30, once. That is the real cost of a full-scale verdict, and it is paid overnight.
• In production: granite stays, definitively for this cycle. The full-scale verdict decides, not our affection for the thesis. Two switching gates, two refusals, zero production regression during the whole campaign.
• The factory is reusable: generating targeted synthetic queries, mining hard negatives on the real corpus, contrastive fine-tuning, anti-leak guard, external judge, full-scale switching gate. It is a generic recipe for adapting an embedder to a domain, applicable to any vertical and any base model.

10. The limits, without makeup

• Our domain bench is n=56 (+ 24 out-of-corpus queries): 1-2 hit gaps are noise there. That is why MIRACL (n=343) judges externally, and why we give both.
• The §2-5 evals are pooled (5,000 passages): absolute values are optimistic, and §6 shows that even relative comparisons between different architectures can invert there. The final judge must be full scale, and it alone.
• The full-scale judge was rebuilt mid-campaign (the original was lost, see §8): the §6 and §8 numbers come from two different judges and do not compare with each other. Each table is internally consistent; we refused cross-comparisons. It is, incidentally, a cross-comparison that made us believe for an hour in a round 4 regression.
• MIRACL-fr is monolingual French: it tests generalist retention, not generalist cross-lingual retrieval. And round 4 was not re-run on it.
• The round 3-4 domain queries are synthetic: they teach the corpus's vocabulary, not the distribution of real user questions. Their coverage was uneven (544 MDN queries versus 49 man pages), and the round 4 autopsy shows the failures concentrate exactly where coverage was missing.
• The fine-tune only ever touched 16.3M of the 144M parameters (head + last 2 blocks). The measured ceiling is that of this recipe, not of the model.
• granite-107m remains the production choice, ours included, and without regret: 107M, Apache-2.0, local, fast. The RWKV thesis keeps the refusal crown and a research program; it has not yet earned the right to carry production.

11. Reproducing

• Base model: howard-hou/EmbeddingRWKV (Apache-2.0). Our pure-torch CPU port is published on Codeberg, EOS fix included.
• Data: mMARCO (unicamp-dl), CodeSearchNet, MIRACL, all public.
• Training: Kaggle free tier. The campaign notebooks and the evaluation harness (56 queries + 24 out-of-corpus) live in Morrigan's private repository; we gladly share them on request, write to us.

Acknowledgements and credits

This work stands on the shoulders of EmbeddingRWKV (howard-hou) and the RWKV-7 "Goose" architecture (BlinkDL and the RWKV community). Comparisons use the excellent granite-embedding (IBM), multilingual-e5 (Microsoft) and EmbeddingGemma (Google). Data: mMARCO (UNICAMP), CodeSearchNet (GitHub), MIRACL. Synthetic generation: Qwen2.5 (Alibaba). Compute: Kaggle free tier.

---

Morrigan is part of the ScarletWolf ecosystem: sovereign AI tools, built to run on your machines, under your control. The full detail of the campaign, including everything that did not work, is in our logs. A result without its failures is an advertisement, not a result.