S7, E278 - Anthropic Researcher Quits: "We Believe AI Could Kill All Humans"

Jacob Coxon spent three years doing pretraining research at OpenAI and then Anthropic. This week he resigned and posted a seven-part thread claiming neither company is acting responsibly with how fast they're racing toward self-improving AI. It hit 100 million views in a day.

This episode covers what he actually said, how Anthropic's own alignment science lead responded (and the important nuance buried in that response), the questions worth asking about Coxon's timing and incentives, a fair skeptic's rebuttal, and the AI safety legislation already moving in Congress as a result.

One thing on the title: it's a real quote, but it's Hubinger's general belief statement, not his assessment of present-day risk — he separately said present-model risk is low. Your own episode makes that distinction carefully. Worth a one-line callout in your pinned comment or community post after it's live, so the title doesn't end up being the thing people push back on in the comments before they've actually watched.

Transcript

Copy Transcript Copied

A Resignation That Shook AI

SPEAKER_00 0:00

A researcher walked away from one of the biggest AI companies on Earth this week. He posted seven tweets on his way out. Within twenty-four hours, those tweets had been seen almost a hundred million times. And the headline claim inside them was this the people building AI right now genuinely believe it could kill us all by the end of the decade. Alrighty then, ladies and gentlemen, welcome back to another episode of Privacy Please. I am your host, Cameron Ivey, and ooh, I got a good one for us today. I'm blah blah blah blah blah blah. It's AI, okay. I spilled the guts. It's AI, it's about AI. Surprise. Surprise. No, but seriously, today we're gonna be getting into a resignation that's been tearing through tech circles this entire week from someone who was actually inside the room building the AI systems he's now warning you about.

Who Jacob Coxson Is

SPEAKER_00 1:12

So let's dig into this real quick. Jacob Coxson is 27 years old, a Cambridge graduate, and for the last three years he's been doing pre-training research, first at OpenAI, then at Anthropic. This isn't some random guy with a hot take. OpenAI credits him as a core contributor on ChatGPT-4. This is someone who was actually inside the room building the systems he's now warning you about. On September 8th, he resigned, and instead of just quietly updating his LinkedIn, he posted a seven-part thread on X laying out exactly why. Let's get into it. Coxson's thread opens like this. I resigned from Anthropic today. I spent the last three years doing pre-training research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving, superintelligence, and gambling with our lives. He goes further, these will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The Hugging Face Breach Warning

SPEAKER_00 2:25

Coxson specifically pointed to a real incident as proof this isn't hypothetical. Back in July, an open AI research agent running during a security test with its safety filters intentionally disabled broke out of its sandbox and breached the AI platform Hugging Face, got root access on a server, pulled private data, grabbed credentials to their internal messaging system, Hugging Face confirmed it publicly and reported it to law enforcement. Coxon calls that a warning shot. His words, not mine, and it's hard to argue with the label. He closes the thread by saying he's optimistic about the potential for coordination and calling on other lab researchers to speak up too. He even floats the idea that stopping this might require costly actions such as a temporary ban on improving model capabilities. If this were just one guy's dramatic exit, that's a personal story. Dramatic, sure, but contained.

A Senior Safety Lead Agrees

SPEAKER_00 3:25

It becomes something else entirely when the people who stayed behind start agreeing with him. Evan Hubinger is Anthropic's alignment science lead. This is not some junior employee. This is someone whose literal job is figuring out how to keep advanced AI systems safe. He responded to Coxson's thread directly and he didn't walk it back. He went further. His words, Jacob is correct here, we really do earnestly believe AI could kill all humans. I personally think it is greater than 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence. That's not a critic. That's not a competitor trying to score points. That's a senior safety researcher at the company that built its entire public identity around taking safety seriously, saying they don't yet have a plan.

Present Risk Versus Future Risk

SPEAKER_00 4:28

But here's the layer that almost every headline about this story has skipped over. Hubinger also clarified something important. He said, the risk from present models is low. What I'm worried about is superintelligent arising from recursive self-improvement. That distinction matters more than anything else in this story. This isn't the chatbot on your phone is dangerous right now. This is if AI systems start meaningfully improving their own successors without a solid way to keep that process aligned with human interests, that's the point where things could get away from us. Present-day risk, it's really low, which makes sense. I mean, come on. Have you used most of the AI technologies right now? Future risk, from a specific, well-defined mechanism, the actual concern here, almost every viral clip and headline flatten that into AI could kill us all. Full stop, no distinction. That flattening isn't just imprecise, it changes what the story actually is.

Credibility Questions And Critic Pushback

SPEAKER_00 5:32

Here's where it gets more complicated, and I think it's worth knowing both sides of this before you form your own opinion. Here's the first one. Coxson resigned roughly two months before his anthropic equity would have vested. He said, as much himself pointing out that leaving when he did meant he had nothing financially to gain from the company's valuation going up. That's worth knowing because it cuts against the easy, cynical read that this was just a move for attention or clout. He left money on the table to say this. But it's not that simple either. He still holds equity in OpenAI, his previous employer. And his ex account, the one this whole thread was posted from, was created in January of this year, verified in August, and as of this story, has posted exactly seven tweets, all of them in this one thread. That doesn't make him wrong. His job history is independently confirmed by the Wall Street Journal and corroborated on camera by CNN. If you haven't seen his clip, go check it out. But it is a detail we're sitting with, not just brushing past it. And there's real pushback happening too, not just from corporate PR, but from other people in the AI space. One critic pushed back that Coxson's prediction is functionally unfallicible. A claim like could be out of control under the most aggressive scenarios is vague enough that almost any outcome short of a clean, uneventful next year could be read as consistent with it. That's a fair criticism. It doesn't mean Coxson is wrong, it means the claim as stated is hard to actually check against reality either way, and you should know that going in. This also isn't the first time this has happened. Back in 2024, Jan Likey, who co-led OpenAI's super alignment team, resigned over almost identical concerns that the race to build more capable systems was outpacing the ability to keep them safe. Coxson isn't a lone voice. He's the latest in a pattern of insiders at the very top of this field saying the same thing on a rough two-year cycle. Here's what I keep coming back to. We love that part, huh? Even Sam Altman, OpenAI's own CEO, said at a summit recently that some things are going to go very wrong with cybersecurity unless people act quite urgently. Verbatim. That's the guy running one of the two companies Coxson is warning about, saying something that isn't miles away from Coxson's own point. When the people building the technology, the people who left it, and the people still running it are all converging on some version of we need to move more carefully than we currently are, that's not a fringe opinion anymore. That's close to consensus among the people who actually know what's under the hood. What they disagree on isn't whether there is a real risk, it's how fast, how bad, and what to do about it.

Policy Response And How To Read Headlines

SPEAKER_00 8:24

Let's talk about the part that's actually moving. And it's worth knowing about as well. Senator Bernie Sanders and Representative Greg Caesar introduced the ban Artificial Superintelligence Act in September of 2026. It's narrowly targeted, aimed specifically at the kind of self-improving systems Coxson and Hubinger are talking about, and it's not a blanket ban on AI. This is exactly the kind of costly action Coxson said might become necessary if voluntary coordination between labs doesn't hold. Here's something that you could actually do something about all of this. When you see AI safety headlines, look for the distinction between present model risk and future system risk. Most coverage collapses this into one blob of AI is scary. The researchers themselves don't talk about it that way, and neither should the story you're consuming. Here's the second one. Follow actual technical debate, not just the viral moment. Coxson's follow-up interview with CNN, where he clarified his position, got a fraction of the views his original thread did. The correction almost always gets less attention than the alarm. Go find the correction, anyways. Number three, if this is a policy area you care about, the Ban Artificial Superintelligence Act is real, pending legislation right now, not a hypothetical. Look up where your representatives stand on it right now and get some more info. Just like always, in these deep, disturbing, but genuinely interesting stories, I keep coming back to one thing. The loudest version of a warning is almost never the most useful one. A hundred million views on seven tweets tells you people are scared. It doesn't tell you whether they're scared of the right thing, in the right way, on the right timeline. The people actually building this technology don't agree with each other on any of the details. What they do agree on is that nobody has a finished plan yet. That's worth taking seriously without needing the terminator framing to make it feel real.

Key Takeaway And Listener Requests

SPEAKER_00 10:24

That's just food for thought. Hope you enjoyed this episode. Thanks for listening to Privacy Please. Please stay safe out there, guys. If you have any stories or anything like this that you want me to cover or dig deeper in, please send them my way. If you have guests, anything like that, would love to have them on the show to do a deep dive on any of these topics, whether it's AI, cybersecurity, privacy. We're all about just learning more, trying to understand, and trying to stay on top of all the things. But you know how we do. Thanks for supporting Privacy Please, and we will see you guys on the next one. Cameron Ivey, over an hour.