Squirrels Have 20 Ears: An Ode to the 1-Bit Quant

I spent an unreasonable amount of my Sunday reading a Reddit thread about someone running a 1-bit quantised version of a large language model on 8GB of VRAM, and I have no regrets. None. The thread is a monument to a very specific kind of joy: watching something extremely sophisticated fail in an extremely stupid way.

For anyone who hasn’t fallen down this particular hole: quantisation is how you shrink a language model so it’ll fit on modest hardware. You take all those billions of parameters, stored normally in 16 or 32 bits of precision, and you squash them down. Down to 8 bits, down to 4, down to 2. And then, because someone always has to see how far a thing can go before it breaks, down to 1 bit. One bit. A parameter can basically only be “a bit yes” or “a bit no.”

The results are magnificent nonsense. Someone in the thread shared output from a heavily quantised TinyLlama answering “10 cool facts about rabbits,” and it informed them that rabbits have four eyes, twenty ears, and a special kind of urine called “dung beetles.” Crows, apparently, have irises that turn red when angry and were considered cursed by the medieval Church. None of this is true. All of it is delivered with the calm, unblinking confidence of a nature documentary narrator. That’s the bit that gets me, not that it’s wrong, but that it’s wrong with such total sincerity.

There’s a genuinely useful technical point buried under the jokes, though, and it’s worth pulling out. One commenter who works on the quantisation side laid out actual numbers: at 1-bit, accuracy on long agentic tasks was around 8%. At 2-bit, 21%. At 4-bit, 68%. The model doesn’t fail gracefully, it falls off a cliff. And the longer the conversation runs, the worse it gets, because errors compound. Ask it one thing and you might get a coherent-ish sentence. Ask it to hold a plan together for thirty steps and it starts calling tools that don’t exist, or the same tool nine times in a row, like a man stuck in a revolving door.

I’ve spent a fair chunk of my career debugging systems that behave confidently while being subtly, catastrophically wrong, so none of this is unfamiliar territory. It’s just usually a misconfigured load balancer, not a robot inventing squirrel anatomy. There’s something almost comforting about a failure mode you can actually see. Half the reason enterprise AI rollouts go sideways is that the failures are quiet: a slightly wrong number in a spreadsheet, a subtly biased recommendation. Nobody’s going to mistake “squirrels have a larger beard than females” for a trustworthy fact. It announces itself. I’d take a loud, obvious idiot over a plausible-sounding one, most days.

The other thing that struck me, reading through, was the argument about why anyone bothers making 1-bit quants at all if they’re this broken. One person called it a waste of electricity and disk space, and I get where they’re coming from, I do think about the footprint of all this constantly, the water, the power, the sheer scale of infrastructure being built to run models that then get squashed down to run a fantasy chatbot on someone’s gaming laptop. But someone else made a fair counterpoint: models trained natively at low bit-depth, like the “ternary” or QAT approaches, are a genuinely different and more promising thing than just chainsawing an existing model down after the fact. It’s the difference between someone born without sight adapting brilliantly, and someone getting blindfolded and told to keep driving.

None of this resolves neatly, and I don’t think it’s supposed to. I’m fascinated by what these models can do and unsettled by the cost of doing it, and both of those things are true on the same afternoon, sometimes in the same sentence. What I do know is that I laughed properly, out loud, alone in my home office, at a fake fact about dung beetle urine. In a week of otherwise fairly grim news, I’ll take it.