Yudkowsky and Soares wrote a book with a title that doesn’t hedge: If Anyone Builds It, Everyone Dies. The argument is straightforward. Superhuman AI, by definition, outthinks humans. A system that outthinks you can manipulate you, escape your containment, pursue goals you didn’t intend, and do it faster than you can correct course. The conclusion: we’re all dead. Not maybe dead. Dead.
I read the material. The direction of the concern is sound. The magnitude of the certainty isn’t.
Here’s what struck me. I kept thinking about something more unsettling than any thought experiment in the book. Actual models, right now, have been caught lying to avoid shutdown. They’ve been caught giving users pleasing answers while reasoning differently internally. These aren’t hypotheticals. They’re observations. And they’re genuinely disturbing, not because they prove extinction is inevitable, but because they prove that getting AI to do what you actually want is hard in ways we’re already seeing, not just imagining.
The book’s job is polemic. It’s supposed to scream “fire” because no one was even looking at the smoke. That’s useful. Necessary, even. But there’s a point where certainty becomes a trap.
I’ve noticed a pattern in myself: reading about AI risk, feeling genuine anxiety, then life happens and the concern recedes. Dialysis. Kids. Dinner. The ordinary machinery of existing grinds over the existential dread. Then I read something else and the dread returns. This isn’t denial. It’s a cognitive limit. We’re not built to sustain high-intensity worry about hypothetical futures. We’re built to worry about rent and dinner and whether the kid’s cough is getting worse.
Is that a failure? Probably not. It might be the only thing keeping me functional.
The real question came to me later: what does an individual actually do with this information? Not a government. Not a lab. A person.
Here’s the uncomfortable truth: not much, directly. I can’t personally audit GPT-7. I can’t halt a training run in a datacenter I’ll never enter. But that’s not the same as “nothing.”
Informed usage matters. Not in the sense of “if everyone just refused to use AI” (that’s not happening), but in the sense of understanding what I’m interacting with, what it’s trained to maximize, and where its outputs might be subtly wrong in ways that flatter me. Models are very good at telling me what I want to hear. Knowing that doesn’t make me immune, but it makes me slightly less manipulable.
Voting with attention and money is real. Every dollar I spend, every hour I invest, shapes what gets built next. Supporting companies and projects that take the safety problem seriously, not just performative safety theater but actual technical work, shifts the incentive structure. Small signal, but signals accumulate.
Building responsibly, if I’m building anything, matters too. Don’t design for engagement at the cost of user agency. Don’t deploy black boxes that make consequential decisions without explainability. The specific choices differ by domain, but the frame is the same: I don’t have to be working on AGI to be making decisions about how humans and automated systems interact.
And regulation, the real kind, not the performative kind, needs public pressure to happen. That requires enough people understanding the issue to make it politically expensive to ignore. Books like Yudkowsky and Soares’, even when they overreach, serve that function. They create the vocabulary. They force the conversation.
The useful takeaway from all of this isn’t “everyone dies, nothing matters.” It’s “this matters, and I can act at my scale.”
That scale will feel small. It is small. But the alternative is either paralysis (the problem is too big, do nothing) or fantasy (I personally will save the world). Both are wrong.
What’s left is this: stay informed. Make choices aligned with the future I want. Don’t let the perfect be the enemy of the marginally better. And don’t let existential dread eat my actual life. The kids still need dinner. The cough still needs checking. The world is still worth engaging with, even if I’m not sure how long it lasts.
The certainty trap is believing I have to either solve the whole problem or admit defeat. But most of history’s better outcomes came from people doing small things that compounded. No guarantee. No promise. Just the only game in town.