Morrigan is our post-LLM AI assistant: it runs entirely on a modest machine, with no GPU and no call to an external service. Generation runs on a quantized RWKV-7 World 2.9B model (a linear-attention RNN, not a transformer) in q4_k_m, and its answers are grounded on a document corpus through strict RAG: with no reliable context retrieved, the model is never called and Morrigan answers "I don't know". Retrieval is therefore not an implementation detail, it is the quality ceiling of the whole assistant.
This week we rebuilt that engine. In one sentence: embedder chosen on a benchmark (a 107M model beats far larger ones), CPU port of an RWKV embedder released open source, two measured negative results, and a multi-corpus architecture. Here is the method and the numbers, failures included, because on this terrain a clean negative result is often worth as much as a positive one.
Our constraint is unusual. Questions come in French, the technical documentation is in English. So we need cross-lingual retrieval (a FR query must retrieve an EN passage) on a domain with sharp vocabulary: function names, error messages, API signatures. An embedder's ranking on a generic English-language leaderboard says nothing reliable about this task. So we benchmarked on our queries.
The metric is hit@k: the share of queries whose correct document appears in the top k results. The evaluation set has 56 FR queries grounded on a code-documentation corpus (Python, JavaScript, CSS, HTML, SQL/PostgreSQL, and shell and git man pages), plus 24 out-of-corpus queries to measure refusal, the "0 hallucination" guard.
A methodological point we own: to compare eight models on a weak machine in one night, we re-encode a fair subset of 5,000 chunks (target documents guaranteed present, plus distractors), identical for every model. The relative hit@k between models is therefore reliable; the absolute figure is optimistic, since the subset guarantees the targets are present. We say so rather than hide it. A repro detail for anyone who wants to rerun the harness: the shootout hangs on a Hugging Face Hub network check even offline, you have to force HF_HUB_OFFLINE=1.
Results, hit@3 out of 56, FR→EN code retrieval:
| model | size | dims | hit@3 | license |
|---|---|---|---|---|
| granite-embedding-278m-multilingual | 278M | 768 | 55/56 | Apache-2.0 |
| granite-embedding-107m-multilingual | 107M | 384 | 53/56 | Apache-2.0 |
| multilingual-e5-large | 560M | 1024 | 50/56 | MIT |
| multilingual-e5-small | 118M | 384 | 48/56 | MIT |
| embeddinggemma-300m | 300M | 768 | 48/56 | Gemma |
| multilingual-e5-base | 278M | 768 | 46/56 | MIT |
| EmbeddingRWKV-0.4B | 389M | 768 | 30/56 | Apache-2.0 |
| EmbeddingRWKV-0.1B | 144M | 768 | 26/56 | Apache-2.0 |
The verdict is clear: granite-embedding-107m (IBM, Apache-2.0). At 107M parameters and 384 dimensions, it beats models two to five times heavier, its weights are local and its license permissive, and its CPU encoding speed matches e5-base. The 278m gains two more points, but for 2.6 times the weight, vectors twice as large, and markedly slower CPU encoding (measured under 0.7 chunk/s versus about 2.7 for the 107m). Two hits out of 56 are not worth that price. We kept the 107m and re-indexed the code corpus with it.
A word on the two RWKV rows, because the number deserves an honest note. The harness results file records the 0.1B at 18/56; the 26 shown above is the same measurement with the asymmetric query instruction (the Instruct:/Query: prefix), now on by default in our engine. The 18→26 gap shows how much that prefix matters for these models. The 0.4B at 30/56 comes from a one-off run, not written back to the results file: measured, but less reproducible than the rest of the table. We come back to it right away, because it is the heart of the matter.
Our underlying bet is architectural coherence: if generation runs on a linear RNN, retrieval should be able to as well. So we wanted to seriously evaluate EmbeddingRWKV, an embedding model built on the RWKV-7 "Goose" architecture.
First obstacle: the reference code for EmbeddingRWKV is CUDA only. It compiles a .cu kernel at load time and forces device="cuda" in half precision. On a machine with no GPU, it cannot run.
So we ported the text-inference path to pure PyTorch, on CPU, with no CUDA dependency at all. The core of the work was to reimplement the recurrent WKV-7 operator and to check that it stays numerically identical to the original kernel, not an approximation. Setting u = w0 + tanh(xw·w1)·w2, our pure-torch path computes the decay like this:
# u, then -softplus(-u) - 0.5
w = -F.softplus(-(self.w0 + torch.tanh(xw @ self.w1) @ self.w2)) - 0.5
# decay applied as a nested exponential
w = torch.exp(-torch.exp(w)) # = exp(-exp(-softplus(-u) - 0.5))The CUDA kernel, for its part, applies w = exp(-0.6065 · sigmoid(u)), with 0.6065 = exp(-0.5). The two expressions are equal, by the identity exp(-softplus(-u) - 0.5) = exp(-0.5) · sigmoid(u). The port is therefore faithful by construction. Engineering honesty: this equivalence is an algebraic identity, proven on paper and checked by a relevance sanity-check, but not yet locked down by an automated test in the repo.
We released this port as open source, under an Apache-2.0 license, as a standalone repository, so anyone can run this embedding model on a CPU. It is a concrete contribution to the RWKV ecosystem: what existed only in a GPU version now runs everywhere. It is also announced and open for discussion on Hugging Face, on the original model's page.
A performance note useful to others. The RWKV recurrence is memory-bandwidth bound on its internal state [B, H, N, N], not compute-bound. Compiling it with JIT barely helps. The only lever that pays off is a small batch (4) that keeps the state resident in cache; a larger batch spills it to RAM and gets slower.
Once the model was usable, the measurement is unambiguous: 26/56 for the 0.1B, 30/56 for the 0.4B, versus 53 for granite-107m. And above all, size does not explain the gap. The 0.4B, larger than the best transformers on the bench, gains only four points over the 0.1B and stays twenty-three points behind granite. This is not a capacity problem, it is a domain problem: this model never saw cross-lingual retrieval over code during training. The score it gets on English-language leaderboards does not transfer to our cross-lingual case.
A concrete symptom: the similarity distribution. A correct match comes out at a cosine of 0.25 to 0.41 on the RWKV, versus 0.75 to 0.95 on granite or e5. The signal is there but compressed, which forces a gate threshold specific to the model (we moved from 0.84 with e5 to 0.64 with granite; the RWKV would ask for yet another).
Assumed conclusion: as a retrieval bi-encoder, we set it aside. Architectural coherence does not buy back a twenty-to-thirty-point drop per answer. The CPU port, though, stays: it is a reusable sovereign artifact, and the option of fine-tuning it on our domain remains open for later.
We also tested hybrid search, which combines the semantic (dense embeddings) and the lexical (BM25, a score based on exact word matching). It is a proven technique, often a winner. We wanted to verify it, not assume it. This bench runs on the full code index, about 46,000 chunks, not on the 5,000-chunk subset; granite therefore scores 50/56 here rather than 53, consistent with the subset's optimistic gap.
| method | hit@1 | hit@3 |
|---|---|---|
| dense (granite alone) | 43/56 | 50/56 |
| BM25 lexical alone | 12/56 | 19/56 |
| RRF fusion (dense + lexical) | 40/56 | 50/56 |
Lexical alone collapses (19/56), and fusing it with the dense adds nothing: hit@3 stays identical and hit@1 drops. The reason lies in the cross-lingual case: a French query shares no words with an English document, except on shared code identifiers. BM25 has almost nothing to bite. Monolingually the conclusion would probably be the opposite; here, we do not build a hybrid, and we decided it by measurement rather than by principle.
That leaves the organization. Morrigan has several corpora: code documentation, general knowledge, and now computer networking. The lazy solution would be to melt everything into a single index. We chose the opposite: keep each corpus in its own index and fuse them at query time.
The benefit is twofold. Maintenance stays independent: we re-encode or replace one corpus without touching the others, and we can evaluate each one separately. And precision is preserved: a code question is not diluted by an encyclopedic article.
The FederatedRetriever searches each corpus, applies a per-corpus relevance threshold (gate = 0.64, the "0 hallucination" guard), then designates the primary corpus by its best score. Another corpus is only summoned if its score stays within a margin of the primary's:
tops[c] = max(tops.get(c, 0.0), s) # best score per corpus
primary = max(tops.values()) # primary = the best score across all corpora
elig = [x for x in cand if tops[x[2]] >= primary - self.margin] # margin = 0.10That is where cross-completion happens, when a piece of information lives in two corpora. Each chunk carries labels (corpus, doc_type, language, topic) that feed a light routing boost (0.05): a question mentioning "in python" or naming a tool (tcpdump, iptables) surfaces the matching documents. A detail that matters for coherence: this boost only reorders and selects; the returned score stays the raw cosine, so the downstream gate keeps its meaning.
We tuned the margin by a sweep. A measured, slightly counter-intuitive result: widening the margin never degrades code precision (it stays stable around 50/56 across the whole sweep), because it is the scores that protect routing, not the margin. And cross-completion stays a minor effect (two to three queries out of twelve in a mixed set), simply because our corpora overlap little. We say it plainly: here, the real value of federation is "answer both domains, well routed", not a rich cross-completion we would have fantasized about.
A sovereign detail on the network corpus: its 612 chunks (sixteen tools, from ip to tcpdump) are built from the machine's local man pages. No cloud dependency, no scraping.
For anyone who wants to redo or reuse this work:
EmbeddingRWKV is a standalone repository, under Apache-2.0: codeberg.org/scarletwolf_ai/embedding-rwkv-cpu. PyTorch and the Hugging Face checkpoint are all you need to run it on CPU, with no GPU or CUDA.ibm-granite/granite-embedding-107m-multilingual (kept), the intfloat/multilingual-e5-*, google/embeddinggemma-300m, howard-hou/EmbeddingRWKV.If you work on cross-lingual retrieval, two points from this work will save you time: lexical does not help when query and document are in different languages, and an embedding model strong on English-language leaderboards can collapse on a domain it never saw in training. Measure on your task.
granite-embedding-107m (sovereign, Apache-2.0, better than far larger models).The thread tying all this together is the same demand as on the rest of our sovereign tools: we measure before we choose, we own what fails, and we give back what we take from the ecosystem.
Because the gain did not justify the cost. The granite-278m scores 55/56 against 53/56 for the 107m, that is two hits out of fifty-six, at the price of 2.6 times the weight, vectors twice as large, and markedly slower CPU encoding. On a machine with no GPU, indexing speed and memory footprint matter as much as the score. The 107m was the best measured trade-off.
Yes, by construction. The decay of the WKV-7 operator is written exp(-exp(-softplus(-u) - 0.5)) in our pure-torch path, which is algebraically equal to the exp(-0.6065 · sigmoid(u)) of the original kernel (with 0.6065 = exp(-0.5)). It is an exact identity, not a numerical approximation. The honest caveat: this equivalence is proven and checked by a sanity-check, but not yet locked down by an automated test in the repo.
No, the conclusion is scoped to our case. In cross-lingual retrieval (FR query, EN document), BM25 has almost no shared words to exploit, so fusion does not help and can even degrade hit@1. Monolingually, where query and document share vocabulary, hybrid very likely wins again. We did not build a hybrid here because our task is cross-lingual.
For two concrete reasons: maintenance and precision. Separate indexes are re-encoded, replaced and evaluated independently, without breaking anything else. And score-based routing prevents a code question from being diluted by an encyclopedic article. Query-time fusion recovers the best of both worlds: each corpus stays provable on its own, and a multi-domain question still gets a well-routed answer.