Why AI Coding Tools Feel Like a Miracle for Some and a Dud for Others

Someone posted a link to their own academic paper this week, analysing two decades of open-source commit data, and it’s been sitting in the back of my head since. The finding: big, mature projects grow at a steady, boring rate no matter what tech hype is happening around them. Smaller projects are chaotic, prone to bursts and stalls. And even with LLMs in the mix through early 2025, the mature projects didn’t show a bump. Nothing. Steady as she goes.

That tracks with something I’ve noticed at work but never quite put into words. I’ve spent the last few years bouncing between a legacy platform with a decade of accumulated decisions baked into it, and smaller greenfield tools we spin up for internal use. On the small stuff, Copilot or Claude feels like magic. I describe a function, it writes most of it, I tidy the edges, done. On the legacy platform, the AI is still useful, but it’s more like a slightly eager junior dev who hasn’t read the seven-year-old ticket that explains why we don’t do it the obvious way. Half my job on that codebase was never typing. It was knowing which of the four PaymentService classes was the real one.

Someone in the comments made the point plainly: developers don’t spend most of their time coding, they spend it on collaboration, context, and the parts of the job that happen at a whiteboard or over a coffee. Another commenter dropped a reference to an actual randomised controlled trial, and this is the bit that really got me. Sixteen experienced developers doing real tasks on mature projects, AI allowed or disallowed at random. Before the study, they predicted AI would make them 24% faster. Afterwards, they still believed it had made them 20% faster. The actual measured result: AI made them 19% slower.

Sit with that for a second. Not just wrong, but wrong in both directions, before and after, with no correction even after living through the experience. That’s not a criticism of those developers, it’s a genuinely humbling data point about how bad any of us are at judging our own productivity when a shiny tool is involved. I include myself in that. I feel faster with an AI assistant open in my sidebar. I have zero rigorous evidence that I am, on the codebase that matters most.

Where I land, tentatively, because I don’t think this is settled: the productivity gain from these tools might really be a function of project scale and the weight of organisational history sitting on top of the code, not a property of the tool itself. Small project, few constraints, few stakeholders, the AI looks brilliant. Large project with three months of approval process for a one-line change, as one commenter put it, the AI can write the line in ten seconds and it still takes three months. The bottleneck was never typing speed.

There’s a broader thing here too, about who benefits from the “AI makes everyone 10x more productive” narrative. One comment cut straight to it: the value doesn’t necessarily flow to the coder, it flows to whoever is deciding headcount. I don’t think that’s a cynical take, I think it’s just an accurate one, and it’s worth remembering next time a productivity stat gets waved around in a budget meeting.

I don’t have a tidy answer for where this settles. I’m still going to use these tools, they’re genuinely useful for the right kind of problem, and I’ll keep updating my opinion as better studies land. But I’m a lot less confident than I was six months ago that “productivity” is even the right word for what’s being measured.