Seventeen Grand and a GPU Bottleneck: The Local LLM Arms Race

I fell down a Reddit hole yesterday, the way you do, and ended up on r/LocalLLaMA reading a thread about someone running a 256GB Mac Studio with an M5 Ultra chip, trying to squeeze usable speeds out of a model called GLM-5.3-Flash. Not exactly light bedtime reading. But stick with me, because underneath the acronyms there’s something genuinely interesting going on, and it says a fair bit about where we’re all headed with this AI business.

The gist: this person has a very expensive Mac, and they’re happy with it, but they’ve noticed the GPU is underpowered relative to all that RAM. So they’re wondering out loud whether a 512GB version would even be worth it, because the bottleneck isn’t memory, it’s compute. Someone else chimes in with two of these machines networked together, getting barely 10 tokens a second on a bigger model, and someone calls it “unusable.” Then a bloke turns up with two Nvidia DGX Sparks and his own custom C++/CUDA inference engine he built from scratch, benchmarking prefill and decode speeds like he’s tuning a race car. One commenter is running four connected units and doing the maths on whether it’s cheaper than a rack of actual server GPUs (spoiler: sort of, depending on what you value).

What struck me wasn’t the tech specs, it’s that these are hobbyists. People spending fifteen, seventeen grand of their own money on hardware to run large language models in their spare room, purely so the model stays on their machine and not on someone else’s server in Virginia or wherever. There’s a real DIY spirit to it, the same energy as blokes who used to build their own PCs to squeeze another ten frames out of a game, except now the stakes feel a bit bigger than frame rates.

I get the appeal, genuinely. I’ve mucked around with smaller local models myself on a much cheaper setup, mostly out of curiosity and a mild distrust of handing every stray thought to a cloud API. There’s something satisfying about a model answering you with zero network round trip, no terms of service changing under your feet, no company deciding next month that your queries are training data now. It’s the same instinct that makes me want to control my own backups instead of trusting the cloud entirely. Except here the entry price isn’t a NAS from Officeworks, it’s a Mac Studio that costs more than my first car.

And that’s the tension I can’t quite resolve. On one hand, this is what a healthy tech culture looks like: people tinkering, comparing notes, open-sourcing quantisation formats, arguing about tokens per second like footy stats. On the other hand, I keep thinking about the sheer amount of silicon, power, and rare-earth mining behind every one of these home rigs, multiplied by however many thousands of people are doing the same thing globally, on top of the enormous data centres already guzzling electricity to run the cloud versions. We’re building parallel infrastructure for privacy and control, which is a reasonable thing to want, but nobody in that thread was talking about where the power comes from or what the embodied carbon cost of a $17,000 chunk of aluminium and TSMC silicon actually is. I don’t say that to lecture anyone chasing a hobby, I have hobbies with footprints too, but it’s a blind spot worth naming.

There’s also a very Australian angle buried in here that made me smile: someone in the comments mentions “Apples to Apples” as a joke about comparing two different Mac Studios, and someone else replies “I mean, it’s literally Apples to Apples.” Nerds making dad jokes about hardware benchmarks. Some things are universal.

What I keep coming back to is how fast this is all moving and how casually the numbers get thrown around. A GPU cluster that would have been eye-watering enterprise kit five years ago is now a weekend project for enthusiasts on a forum, and the models running on it are already being described as “old” or “the smaller one” within months of release. I don’t know where the ceiling is, or if there is one. I don’t know if the answer to “should there be one giant Mac Studio in every home office” is yes or deeply, deeply unnecessary. Probably both.

What I do like is that people are asking the right questions out loud, in public, for free, instead of taking a vendor’s marketing at face value. Somebody in that thread pointed out a “GPU sleep bug” might be skewing the benchmarks, someone else linked their own open-source inference engine, someone compared real-world numbers to the official ones and found the official ones were lying a bit. That’s the internet working the way it’s supposed to: nerds keeping each other honest. I’ll take that over polished keynote slides any day, even if I’ve got no plans to spend seventeen grand finding out for myself.