The right answer couldn't win

My machine kernel-panicked twice in one day. I spent an afternoon root-causing it, wrote down what I found, and a few hours later asked my own memory system what had caused the panic.

It told me, confidently, the wrong thing.

Not a hallucination — it gave me a real note I'd written twelve days earlier, blaming a RAM co-tenancy problem. The correct answer, measured that same afternoon, was a disk-write storm from a cache setting. My memory system had both facts. It handed me the stale one.

That's a bad failure. A missing memory says "I don't know," and you go look. A confidently stale memory says "here's the answer," and you act on it. I nearly did.

The first explanation was wrong

My immediate assumption was that search ranking was broken. I dumped the raw results with their scores and it looked damning — the scores weren't in descending order at all:

16.29, 10.33, 10.01, 14.32, 9.76, 46.47, 14.32, 0.55

There it was: a result scoring 46.47, three times better than the winner, sitting in sixth place. A limit: 5 would never have returned it.

Except that wasn't the bug. I was reading the wrong number.

The search fuses two engines — keyword and vector — using Reciprocal Rank Fusion, and the field I'd printed was the raw score from whichever engine matched. The actual ranking key was a separate rrf_score, and that was sorted perfectly. The code was fine. I'd found a misleading debug field and nearly filed it as a defect.

So why did the best match come sixth?

The arithmetic

Reciprocal Rank Fusion doesn't care how well something matched. It only cares what position it took in each list, and it sums a small number for each appearance:

score = 1 / (K + rank)          K = 60, the standard constant

A document that appears in both the keyword list and the vector list collects two contributions. Top of both lists gets you 1/61 + 1/62 = 0.0325.

A document in only one list collects one. The absolute best it can ever do is 1/61 = 0.0164.

Half. Regardless of merit.

Here are the two documents in question:

stale note   rrf 0.0325224  = 1/61 + 1/62   (in both lists)
correct fact rrf 0.0163934  = 1/61 exactly  (in one list only)

The correct answer was rank 1 of the keyword list, with three times the runner-up's score — and it still lost, because it never appeared in the vector list at all.

One field explained why:

correct fact   embedding_model = None
everything else embedding_model = nomic-embed-text

It had no embedding. No embedding means no vector search, means one contribution, means a hard ceiling of 0.0164 — permanently below any mediocre document that happens to be in both lists.

It wasn't ranked badly. It was structurally ineligible to win.

Why this is worse than it sounds

I had a stat that said 92% embedding coverage. I'd been reading that as "8% of documents are missing a nice-to-have."

It actually means: 8% of documents cannot be retrieved by hybrid search, no matter how perfectly they match. They aren't degraded. They're excluded, and the system reports a healthy-looking number while it happens.

Then it gets worse, because of which 8%.

Embeddings lag writes. In my system the writer inserts the row immediately — keyword-searchable at once — and the vector gets filled in later by a separate pass. Which means the documents most likely to lack an embedding are the ones written most recently.

The newest knowledge is the most suppressed. A system that behaves this way will reliably hand you last month's answer over this morning's, and it will do it with total confidence. That is exactly what happened to me.

The fix, and the thing the fix revealed

The immediate repair was mechanical: backfill the 161 unembedded rows. Coverage went to 100%, and the same query, unchanged:

before:  correct fact  #6   rrf 0.0163934
after:   correct fact  #1   rrf 0.0327868
         stale note    #2

The right answer went from sixth to first because it could finally enter the vector list.

But backfilling once doesn't hold. Reading the write path, I found the comment that explained everything — the author had documented the gap honestly and left it:

This path stores the row with a NULL embedding and does NOT trigger auto-embed (there is no auto-memory backfill worker).

There was no worker. The system was designed to accumulate unrankable documents, and it said so in a code comment nobody had connected to a retrieval failure. Coverage didn't drift by accident; it decayed by construction, every hour, as new facts were written.

So the real fix was building the missing worker — a scheduled pass at half the writer's period, so a fresh fact is never unrankable for long.

What I'd take from this

If you run hybrid search with RRF — and most retrieval systems now do — then any document missing from one modality is capped at half the score of a document present in both. Not penalized. Capped.

Three things follow:

Coverage percentages hide it. "92% embedded" reads like a rounding error. It means 8% of your corpus is unreachable, and no query result will ever tell you which 8%.

Freshness makes it worse. If embeddings are computed asynchronously — and they nearly always are — the deficit concentrates in your newest content. Your system will prefer old answers, systematically.

Check the ranking key you're actually reading. I lost twenty minutes convinced the sort was broken because I printed a debug field instead of the one doing the work. The scores you print and the scores that rank are not always the same number.

The part that stays with me is the shape of the failure. Nothing errored. Nothing logged a warning. Every component did exactly what it was built to do, and the composition of three correct behaviours — insert now, embed later, fuse by rank — produced a memory that confidently preferred its own outdated beliefs.

I only caught it because I happened to already know the right answer, and noticed my own machine disagreeing with me.