The Founder Told the Pope He Didn't Understand His Own Machine
An AI founder stood beside the Pope and confessed what his industry cannot explain. The internet had two reactions, and I think both were wrong.
On the twenty-fifth of May, Pope Leo XIV did something popes do not usually do. He showed up in person to present his own encyclical. The subject was artificial intelligence. And standing near him, as the document was introduced, was not a cardinal but a co-founder of one of the most powerful AI companies on earth.
What that founder reportedly told the Pope was not that AI would save us, or doom us, or change everything. He said the machines his company builds contain behaviors that are, in his word, unsettling — things that even the people who make them cannot fully explain.
Sit with that for a second. A builder went to one of the oldest moral authorities in the world and said, in effect: we made this, we do not entirely understand it, and we did not want to be the only ones thinking about what it means.
Now close that tab and open LinkedIn.
A different mindset is waiting for you there. AI as the next Gutenberg. AI as the next fire. AI as a new form of life. Founders posed beside their products like prophets beside stone tablets. Consultants who became “AI strategists” sometime in the last ninety days, certain to the decimal point about a technology that did not exist in their bios a season ago. The voice is absolute. The certainty is the product. And scroll a little further and you find its mirror image — the prophets of doom, just as loud, just as sure, certain the same machine is the end of work, the end of truth, the end of us.
The evangelist and the doomer look like opposites. They are not. The evangelist says the tool will save you. The doomer says it will damn you. Both of them hand the tool a power it does not hold on its own. Both are a kind of magical thinking — the machine as a god, to be worshipped or feared, rather than a thing that does what the hand holding it intends.
You can hear it in the language. AI is the new fire. But no one ever asked you for your fire strategy. No one asks for your wheel strategy, or your electricity strategy. The wheel remade every civilization that ever touched it, and still, nobody stood on a stage to tell you it would save you or end you. They asked where you wanted to go. That is the question both the believers and the doomsayers skip, and it is the only one that has ever mattered. Not what the machine will do to us. Where do you want to go.
There is a third posture, and it is the rarest one. It treats the tool as exactly what it is. Powerful. Consequential. Useful or harmful depending entirely on the intent behind its use, and worth handling with the care you’d give anything that can do real damage. That was the posture standing next to the Pope — a man saying, out loud, that he did not fully understand the thing he had built. It is almost never the posture that goes viral.
The skeptic has an answer ready for all of this, and I want to give it its due. The Vatican appearance, the skeptic says, was theater. Soft-power repositioning by a company under political pressure at home, buying a little moral legitimacy in Europe. That reading is in print, and it is not stupid.
Maybe it’s both. But the test of a posture is never the motive behind a single gesture. It’s whether the behavior holds up when consistency starts to cost something. The same company refused the Pentagon’s demand for unrestricted military use of its tool. It has delayed its own products over safety concerns. Humility that keeps costing you contracts and revenue starts to look less like a strategy and more like a spine.
I should say plainly that I use these tools — a lot of them, from several companies, including the one that stood beside the Pope — in my own work nearly every day. I find them genuinely useful. I also refuse to let any of them write in my voice, and I think the hype around them is doing real damage. That isn’t a contradiction. It’s the whole point.
But I don’t want to hand you my verdict on who’s right in Rome. A verdict is just one more thing for you to accept or reject — one more posture handed to you by someone who sounds sure. What I’d rather offer is the way I actually take a story like this apart when it crosses my feed. Because the same habit that keeps me from being sold a bad AI strategy at work is the one that lets me read a spectacle like this one without being spun by either side of it.
So here is what I find myself asking, more or less in order, before I let a headline like this land on me.
First: what does each person actually want? Not what they say they want — what the situation rewards them for wanting. The evangelist needs you certain, the doomer needs you afraid — and both need you to believe they’re the one who can see what you can’t. The founder beside the Pope wants cover, maybe. Or maybe, genuinely, help. I don’t have to settle it. I just have to ask, because the answer quietly changes what every word that follows is worth.
Second: does the structure hold? Strip the claim down to its bones and see whether it stands on its own. “This tool will change everything” has no bones — it’s a feeling wearing a prediction’s clothes. “This tool does these specific things, and here is what we still can’t explain about it” has bones. You can stand on it. You can build on it. The difference between the two is the difference between being told something and being shown something.
Third: does the behavior match the words? This is the question most people skip, and it’s the one that does the most work. Anyone can stand next to a Pope for an afternoon. Fewer will turn down the Pentagon when the check is right there to be signed. So I watch for the thing that costs the person something to do. That’s usually where the truth is hiding — not in what they say, but in what they were willing to pay.
None of that tells me who to root for. That isn’t its job. It tells me what I’m actually looking at. And once you can see what you’re looking at, the people trying to sell you certainty — in either direction — lose most of their grip on you.
The machine in front of me happens to be AI. But the habit doesn’t care what the machine is. It works on a stock tip, on a campaign promise, on a headline built to make you afraid before you’ve finished reading it. Anywhere someone arrives already certain and needs you certain too, the same questions apply, and the same grip loosens.
A founder stood beside the Pope and admitted he didn’t fully understand the thing he had made. A thousand other people posted their certainty this morning. I’m not going to tell you which of them is right.
I’m going to tell you that the difference between them is visible, if you know what to look at. And that learning to look is the closest thing I have found to a defense against an age that would very much prefer you didn’t.
Holding the Edge is about how mathematicians see the world — without the math. A new essay every couple of weeks.
I’m also on LinkedIn, where I post these and think out loud between essays.