One Bit, Five Months, and a Backup Drive That Was Too Full

Someone on r/selfhosted posted a write-up this week about bad RAM. I haven’t stopped thinking about it. It’s the sort of story that makes you want to go and check things, and then makes you a bit sad about what you might find.

The short version goes like this. A Proxmox host, running on a small Lenovo box with non-ECC memory, started kernel panicking. The logs said nothing useful. The panic record, buried in a place most of us never look, showed a pointer that was wrong by exactly one bit. Bit 54. Every crash had the same flipped bit. Once the owner saw that, the driver-bug theory fell apart and the hardware theory took over.

I like this for the detective work. Same bit every time is not a software bug. Software is chaotic in its failures. Hardware is stubborn and repeats itself.

The part that got me

The crashes weren’t the scary bit. The scary bit was what the bad memory had touched quietly. A cached copy of a library with bytes flipped. A kernel module corrupted on disk. A Perl library with 61 bytes changed, all at bit 54. Nothing announced any of it.

And then the point that anyone who has built a backup system should sit with. ZFS checksums, Proxmox Backup Server digests and rsync checksums all protect data after it has been hashed. If the bad RAM flips a bit before the hash, the system faithfully checksums the damage, stores it, and verifies it green every night. For months. The poster found one backup chunk, written through the bad host five months before the first crash, with a single flipped bit. Same bit.

I work in DevOps. I have said the words “we have checksums” in meetings with a straight face. This was a useful reminder of what a checksum actually promises. It promises that what you stored is what you were handed. It says nothing about whether what you were handed was right.

Saved by a failing job

The most human detail in the whole post is that the poster’s rsync job to a USB drive had been failing for a month. The drive was 98% full. TrueNAS had been sending alerts, and they’d been putting off dealing with it. And that accidental neglect meant the drive still held old snapshots from before the RAM went bad, because rsync without --delete never pruned them.

So the system that was broken saved the person who owned it. I find that both funny and a bit terrifying. It’s not a design. It’s luck wearing the costume of a design.

One commenter made the point that pruning old backups aggressively isn’t free, and that an older offline generation is the thing you actually want after an incident. I agree with this and I also know I don’t do it. My retention policy at home is roughly “whatever fits.” I’m mildly proud of how cheaply I’ve built it, which is exactly the problem. A bargain is a bargain until the day you need the thing you saved money on.

ECC, and the price of everything

The poster’s lesson list ends with “get ECC if you can.” Another commenter pointed out, accurately, that server RAM is close to unaffordable at the moment. A third noted that AI data centres are soaking up the supply. So the advice is right and the world is making it hard to follow.

I have mixed feelings about this one. I’m fascinated by what these AI systems can do, and the poster used one to work through the logs and learn where Linux hides its crash records. One commenter called it a very experienced technician for a few cents. That’s a fair description of my own experience with them on a good day. But the same appetite for hardware is why my teenager’s next laptop will cost more, and why a hobbyist can’t buy sensible memory for a home server. I don’t have a tidy resolution to that. Both things are true. The tool is useful and the bill is landing on people who never ordered it.

What I’m doing about it

I’m not going to pretend I’ll rebuild my setup this weekend. I will do three small things.

1. Run a memory test on the home server, which I haven’t done since I built it. Embarrassing to type.
2. Keep one old backup generation offline and stop pruning it.
3. Read the alerts. Actually read them, not just see that they exist.

The poster’s closing line was about reading your NAS alerts, and it’s the best advice in the list because it’s the least technical. Most of the disasters I’ve seen in my career weren’t missing tooling. They were a warning sitting in an inbox.

I did enjoy the ending, though. No data lost, a thorough sweep of 45,000 chunks on separate hardware, and a person who wrote it all up so the rest of us could learn for free. That’s a good corner of the internet, and I don’t say that often.