The bug the gate wouldn't let through

I wasn't auditing anyone's code. I was trying to make my squad faster, and a test I'd written refused to go green.

That test turned out to be the whole story, so let me start there. My agents run on models served from my own Mac, and they use a trick called speculative decoding: a small fast model guesses the next several tokens, then the real model checks all the guesses at once. Checking five guesses in one pass is much cheaper than generating five tokens one at a time, so when the guesses are good you get most of them almost free.

The rule that makes this safe is that it has to be invisible. You're only allowed to accept a guessed token if the real model would have produced that exact token anyway. Done right, speculative decoding changes the speed and nothing else — same prompt, same output, byte for byte.

So I wrote a gate that checked precisely that, with no tolerance at all. Not "close enough." Zero. Run the batch check, run the same tokens one at a time, and require the two to match exactly — the predictions and the model's internal memory of the conversation, every layer of it.

It wouldn't pass. And the way it failed was strange: the output was perfectly repeatable — same prompt, same answer, every time — but it never matched what plain one-at-a-time generation produced. Repeatable and wrong is a much worse smell than random and wrong. Random says "noise." Repeatable says "you are consistently doing a specific thing incorrectly."

Finding it

Rather than read code and guess, I built a small diagnostic that ran both paths side by side and compared the model's internals tensor by tensor, reporting the first place they diverged.

The answer came back immediately and it was oddly specific. Attention has three pieces — queries, keys, and values. The keys were wrong by a lot. The values were identical to the last bit.

That single asymmetry names the bug. Position information in these models is applied by rotating the queries and the keys — and not the values. So if values are perfect and keys are wrong, the content is fine and the position is wrong. Nothing else produces that signature.

Here's what the code was doing:

# Build position ids for each row of a decode window.
starts = starts.reshape(-1)[:batch_size]
if self._uses_minimax_m3_positions:
    steps = mx.arange(seq_len, dtype=starts.dtype)
    return starts[:, None] + steps[None, :]          # positions advance ✅
return mx.broadcast_to(starts[None, :, None], (3, batch_size, seq_len))   # ❌

Look at the two branches. The first walks the positions forward — token one, token two, token three. The second copies the starting position across the whole window.

So when the model checked five guessed tokens, it was told all five sat at the same position — the first one. Every guess after the first was being evaluated as if it were somewhere it wasn't.

And it didn't stop there. When a guess was accepted, its wrongly-positioned memory got written into the conversation's cache and carried forward. The error didn't evaporate at the end of the check; it accumulated.

The fix is boring, which is how you know the diagnosis was right:

starts = starts.reshape(-1)[:batch_size]
steps = mx.arange(seq_len, dtype=starts.dtype)
seq_positions = starts[:, None] + steps[None, :]      # advance, always
if self._uses_minimax_m3_positions:
    return seq_positions
return mx.broadcast_to(seq_positions[None, :, :], (3, batch_size, seq_len))

Twenty lines. It had been sitting in a widely-used open-source engine, and it hid because the branch right above it did the same job correctly — so the file looked like it handled the case.

What I'd take from it

Write the gate with no tolerance. "Close enough" would have passed this bug forever. The failing test was worth more than any amount of reading, because a threshold you can negotiate with is a threshold that eventually gets negotiated.

Make the diagnostic say where, not whether. Knowing the outputs differed told me nothing for days. Knowing that keys diverged while values were bit-clean pointed at one specific mechanism, and the fix followed in minutes.

Repeatable-but-wrong is the dangerous failure. It looks stable. It passes every "is it deterministic?" check. It ships.

And ship the test that fails. I sent the fix upstream with a regression test, and the thing I care about is that the test fails without the patch — assert 50 == 51.0, the second token in the window stuck at the first token's position. A test that can't reproduce the bug is decoration.

It merged. What I keep thinking about is that I never went looking for it. I built a gate to protect my own agents, refused to loosen it when it was inconvenient, and it found a real bug in somebody else's engine on the way.