The hundred gigs we left on the table

We run a 750-billion-parameter frontier model on one Mac. Not a distilled stand-in, not an API — the whole thing, GLM-5.2, open weights, MIT-licensed, sitting on the desk. The model file is 211 gigabytes. The machine has 128 gigabytes of RAM. It runs anyway, because antirez's ds4 streams the weights off the SSD and only keeps the parts it's using in memory — about 15 gigabytes at any given moment. A 211-gig model living in 15 gigs of RAM. That fact by itself is most of why we do this: a frontier model you own, on hardware you own, with no meter running.

But it's slow. About 2.4 tokens a second, where the lighter DeepSeek-V4-Flash does around 6 on the same engine. So I went hunting for the speed.

The first instinct was to hide the waiting. A mixture-of-experts model only fires a handful of its experts per token; if you can guess which ones you'll need next and pull them off the SSD while the GPU is still busy, the wait should disappear. We built the prefetch. It barely moved the number. So I measured the thing I should have measured first: how fast can we actually read from the drive? 14.3 gigabytes a second — and random reads came back as fast as sequential, which means we were already sitting at the drive's real ceiling. You can't prefetch your way past a wall you're already pressed against.

That looked like the end of the software road — next lever would be more drives, more money. And then the actual bottleneck walked in, and it wasn't the drive at all. The engine caps its expert cache at 12 gigabytes. On a 128-gig machine, running a model whose hot experts recur far more than they look like they do at first glance, we were leaving roughly a hundred gigabytes of RAM sitting completely idle — re-fetching experts off the SSD that we had every bit of the room to simply keep.

So the real lever isn't hardware we don't have. It's the RAM we already paid for: lift the cap, give the cache the hundred gigs, and hand each layer its own budget so a greedy one can't trample the rest. We haven't shipped the speedup number yet — I don't publish gains I haven't earned on a benchmark — but that's the hunt, and that's exactly where it's pointing.

The lesson I keep relearning: the first wall you measure is rarely the one you're actually stuck against. We came within a hair of buying drives to solve a problem that was a config default leaving our own memory on the table.