“Nobody Knows Nothing”.

On Thursday, Anthropic posted that they would pause AI development if their competitors did too. Read that twice. A company whose entire existence is a bet on building this thing faster than all others is volunteering to stop. The reason is RSI, recursive self-improvement, the AI getting good enough at building AI that it starts improving itself. They claim 80% of recent Claude code was written by Claude.

The natural response is: Holy Shit. But that treats this as information about AI, which might be the wrong read. What if we looked at it as information about prediction itself. The smartest researchers in the world are looking at what they built and are telling us they don’t know what’s going to happen. (In the same way that they admit they don’t actually know how AI works, but that’s another essay.) I think that should interest us.

So, some people think the right response to dread is a better estimate, a better fit model. And ten years ago when I was reading these guys, the AI safety crowd, I might have bought into that, though at the time it all seemed so far-fetched as to not actually have stakes. Such that a few years later, in April 2020, visiting my parents for the week that turned into a few months, reading news voraciously, learning the meaning of R value and bending the curve and fomites, for more information, for better models, so I could know what was happening, what was coming. And what did that get us? Not nothing, but not much. Grasping for control is an entirely human impulse. It’s why I would feel more comfortable flying the plane myself. But what would that get me? A near 100% probability of dying.

The problem with estimating likelihood is that some events don’t have priors. Unlike plane crashes, we’ve never invented AI before. And the doomer scenarios, fun yet chilling, are a tour of these. Bostrom’s paperclip optimizer: an AI told to make paperclips gets efficient, gets superintelligent, turns the earth and eventually the solar system into an expanding ball of paperclips. That’s the alignment problem, how to keep its goals pointed at ours, like continuing to exist. The escape scenario: the AI gets off its airgapped machine, emails a synthetic biology lab, orders novel DNA, which arrives in the mail and starts a pandemic. Last week OpenAI, DeepMind, and Anthropic, whose leaders hate each other, jointly asked the US government to limit synthetic biology labs now, because the models design DNA better than a grad student. And we attached the AIs to email long ago.

I’ll stop there. Though not before Ray Kurzweil, who I read as a teenager, in his simultaneously thrilling yet vague The Singularity Is Near, imagining a point in the future beyond which we cannot see because we merge with the machines, or are uploaded, or annihilated, unclear. And he is the optimist! This is our event horizon, so named for the rim of the black hole beyond which light cannot escape and we physically cannot see. So what comes after, when AI is smarter than us and starts making its own plans?

You cannot estimate your way harder across that line. And it hurts your brains to try. You see, the human brain is built to project a world from a series of cuts. Your eyes don’t pan smoothly; they fix, then skip, fix and skip. These are called “saccades.” Then the brain stitches the jump cuts into a steady pan, called saccadic masking. And masking works pretty good, we got a few million years out of it! But our brains need somewhere to land. The horizon gives us nothing. Nothing to fix on.

A Hollywood wit said of the industry, nobody knows nothing. Taken one way, it’s a koan. The future is, will be, and always was unknowable. There are no shades of not knowing, it is a line, the veil. Twenty years ago I stood in line at the Egyptian theater for a late night show of Dr. Strangelove, marveling to my friends how back in the day people thought the world might end any minute. How could they marry? How could they have kids?! From behind us an older guy said, “yeah. I was there. It was scary as hell.”