The Mirror, Not the Oracle

Someone on a forum I read asked a simple question this week: what’s one thing AI does surprisingly well that you didn’t expect? I’ve been chewing on it since, mostly because my answer has changed twice in three years.

When ChatGPT first landed, I figured I’d use it like a faster Google. Ask a question, get an answer, move on. That lasted about as long as most of my New Year’s resolutions. What actually happened is I started feeding it my mess: half-formed emails, notes that made sense to nobody but me at 11pm, an argument I was having with myself about whether to switch super funds. And it turns out that’s the bit I didn’t see coming. Not the answers. The clarifying.

One reply in that thread stuck with me, someone using it as a “steel man” debate partner, deliberately prompting it to argue against them rather than agree. I’ve started doing something similar, mostly out of self-preservation. I spend a lot of time in my own head, and like most people who do that, I’m very good at building a case for whatever I already believe. Having something argue back, without the social cost of annoying my wife or a mate at the pub, has been genuinely useful. Not because the AI is right. Because it’s relentless, and it doesn’t get tired of the argument the way a human would.

Another comment described it as “holding up a mirror” rather than generating anything new, using it to catch the gap between what they meant to write and what a reader would actually take from it. That’s close to how I use it for work documentation. I don’t need it to write the thing. I need it to tell me where I’ve assumed knowledge the reader doesn’t have. That’s a genuinely different function to what I expected an AI chatbot to be good at, and it’s the one I now rely on most.

There’s a tension here I don’t think I’ve resolved, and I’m not going to pretend I have. I find this stuff fascinating, properly fascinating, in the way I used to feel about the internet before the internet started making everyone miserable. But I also can’t shake the unease about what it costs to run these things, the water, the power, the data centres humming away somewhere I’ll never see, in a country that still can’t get its own energy transition sorted. I use the tool and worry about the tool in more or less the same breath. I don’t know how to make that neat, so I’m not going to try.

My own daughter uses it for study notes, turning a wall of textbook text into something she can actually revise from. I watched her do it last month and felt two things at once: relief that she’s got a tool I didn’t have at seventeen, and a low hum of worry about what she’s not learning to do herself. Both true. Neither cancels the other out.

The thing I keep coming back to is that nobody predicted this particular use case, mine or anyone else’s in that thread. Not the companies building these models, not the doomers, not the boosters. We all thought we were getting an oracle. What we got, mostly, is a very patient, occasionally overconfident thinking partner that’s better at finding the gap in your logic than at telling you the truth. That’s a stranger and more useful thing than I expected, and I’m still working out what to do with it.