Do two models think alike? We measured it, then had it torn apart

People have been gluing models together since 2023. Take layers from one, experts from another, stitch, publish. The benchmarks went up and the models felt worse, and mostly nobody could say why, because there was no instrument — you found out your stitch was bad by using it for a week.

There is a cheaper question to ask first. Not can these two models be joined, but do they already agree.

Acceptance is the ruler

Speculative decoding works like this: a small model drafts a token, the big model checks it, and if they match you got the token for the price of the check. The useful part is what the match rate means. A drafter never touches the target's internals. It only has to predict its tokens. So the acceptance rate is a cheap mechanical measure of whether two models have converged on the same behaviour — and you can measure it without attempting the graft at all.

That inverts the usual order. Splice, then discover it's broken, becomes measure, then decide whether splicing is even worth trying.

The measurement

Two models from the same family, different sizes. Laguna XS is 40 layers at 2048 wide; Laguna S is 48 layers at 3072. They cannot be spliced — an XS expert maps 2048→512→2048 and an S expert maps 3072→1024→3072, wrong in both dimensions, different tensor naming, different quantisation formats entirely.

They agree on roughly 85% of tokens anyway.

XS drafts for S     0.8474   (2,399 / 2,831)   95% CI 0.8337–0.8602
S drafts for XS     0.8447   (2,595 / 3,072)   95% CI 0.8315–0.8571

Acceptance doesn't care that the tensors are the wrong shape. That is the whole point of using it as a probe.

Then we had it torn apart

The write-up above originally continued for three more paragraphs, explaining that the relationship was symmetric, that this proved the two models shared a representation basis mutually rather than one being a compressed copy of the other, and that the instrument had been validated by an exact identity gate.

None of that survived review.

The work was handed to GLM-5.2 — a non-Claude model, running agentically with shell access, told to find what was wrong and to verify by running things rather than reading them. It made 52 tool calls, re-derived every number from the raw JSON, re-ran the gates, broke them itself in its own way, and opened the files whose contents had been asserted.

Here is what it took out.

"Symmetric" was a null result wearing a finding's clothes. Two proportions, 0.8474 and 0.8447, with overlapping confidence intervals. Overlapping intervals are not a test. The actual two-proportion z is 0.284, and the confidence interval on the difference is [−0.016, +0.021] — straddling zero. That is "cannot reject equality," which is not the same claim as "is symmetric," and the story built on top of it about mutual sharing was narrative, not evidence. Worse, the two numbers come from different token populations — one scores S's generations, the other scores XS's — so they aren't even samples of the same distribution.

The identity gate could not fail. The instrument was checked by having a model score its own greedy output, which should return exactly 1.0000. It did. That was presented as calibration discipline — ask the tool something you already know before asking it something you don't.

Except the scoring path and the generation path are the same computation. Both prefill the prompt into a cache and step one token at a time. Self-agreement was 1.0000 by construction and could never have been anything else. It proved nothing.

The galling part is that it had been a real check. The first version scored with a single full-sequence forward pass — a genuinely different code path — and it returned 0.9883, which is how the near-tie problem got found at all. Switching to the cached path to "fix" the discrepancy also quietly converted the gate into a tautology. The instrument was made to agree with itself and that was recorded as a win.

The shuffled control — permute the target's tokens, re-score, watch it collapse to 0.0404 — is the only check in the harness that can actually fail. It does the work the identity gate was being credited for.

The "noise floor" was diagnosed wrong. Nine positions disagreed between the two scoring paths, all within a few ULPs of bf16 at logit magnitude ~28. That got written up as harness noise, removed. It isn't noise — those are positions where the model is genuinely indifferent between two tokens, and the two paths land on different sides of a coin-flip. Real model behaviour, not instrument error.

And for the cross-model numbers there is no generation to match against, so that ~1% path-dependence is still sitting inside 0.8474. Switching paths didn't remove it; it removed the ability to see it. Quoting four significant figures against a ±0.013 interval with an unacknowledged ~1% systematic underneath is overclaiming.

One number hid the structure. Per-prompt acceptance ranges from 0.646 to 0.906. The probe's own docstring warns that acceptance varies by regime and that a single-domain number would misrepresent the model — and the write-up did exactly that anyway.

Also, quietly: every model load prints a tokenizer warning about an incorrect regex pattern. Both models share the tokenizer so the comparison stays internally consistent, but it should have been disclosed next to a number quoted to four decimals.

What survived

The numbers reproduce exactly. The Wilson interval implementation was re-derived independently and matches to 1e-9. The scoring code has no off-by-one — the prompt span is correctly excluded and exactly one position is counted per generated token. The gates on the game repo genuinely fire: a type error takes tsc to exit 2, a perturbed geometry assertion takes node --test to 7-pass-1-fail by the correct test name, a syntax error takes the build to exit 1.

So the operational claim stands: two models that cannot share a single tensor agree on about 85% of their next-token choices, and you can find that out in an afternoon instead of a training run. That is still the useful result, and it is still enough to decide where a graft is worth attempting.

What doesn't stand is everything that was layered on top of it — the symmetry, the validated instrument, the removed noise floor, the single clean number. Four claims, none of them load-bearing for the result, all of them added because they made a good measurement sound like a better one.

The actual lesson

The instrument was built carefully. It had a calibration gate and a negative control, which is more discipline than most of this genre gets. And the calibration gate was still circular, and nobody noticed until something that wasn't invested in the result went and read the two code paths side by side.

Self-review has a ceiling, and the ceiling is exactly the assumptions you didn't know you were making. The fix isn't more care. It's an adversary with shell access and no stake in the answer.