AI is advancing faster than our ability to understand or control it
Steve Middleton · steveontheweb.com
I’ve worked as a video game developer for the past 15 years
Since 2017 I’ve helped build numerous machine-learning systems, from research prototypes to production tools. Plus ongoing study and personal projects.
I’m speaking as an individual. These are my own opinions, not my employer’s.
// They’re answering two different questions
Cure disease, end scarcity.
Powerful, and we can’t yet make it safe.
◀ I’m hereOverhyped. Keep using it.
Jobs, scams, bias.
I’m in the top-right. So, it turns out, are many of the people building it.
“We live during the most important era of human history.”
Toby Ord, The Precipice (2020). He puts the odds of existential catastrophe this century at about 1 in 6, and from AI alone about 1 in 10.
We are entering “a rite of passage, both turbulent and inevitable.”
Dario Amodei, CEO of Anthropic · Jan 2026
“I would put at least a ten percent chance that it causes human extinction, and that is ridiculously high.”
Neel Nanda, researcher at Google DeepMind · Sep 2026
Anthropic and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.”
Jacob Coxon, resigned from Anthropic · Sep 2026
“We’ve never had to deal with things more intelligent than ourselves before.”
Geoffrey Hinton, Nobel laureate and “godfather of AI”
May 2023: the CEOs of OpenAI, Anthropic and Google DeepMind, along with Hinton and Bengio, signed a statement saying AI extinction risk should be a global priority alongside pandemics and nuclear war.
→ Here’s why I think we’re not ready: look at how fast things have moved over the last year.
// A new Claude model from Anthropic that’s unusually good at hacking software
sandbox = a sealed test computer, walled off from the internet
Not released
Given to defenders first (Amazon, Apple, Google, Microsoft and others) to fix holes before attackers find them
0 found
0 fixed (about 1 in 12)
Finding got fast. Fixing is still human speed.
The sandbox test was instructed: the model was told to escape and message the researcher; the test computer was separate from the machine holding the model. Unprompted: posting its exploit publicly. Apr 7: Mythos Preview announced, not released; shared via Project Glasswing ($100M in credits); 10,000+ high or critical vulnerabilities found by late May. Jun 9: a safeguarded version (Claude Fable 5) goes public; the full model stays restricted. Patch figures as of Oct 2, 2026. Sources: Anthropic system card, CyberScoop, CSO Online, Software Improvement Group, The Hacker News.
Conceptual animatic based on OpenAI’s report and the METR/Redwood investigation (Aug 26, 2026).
agents on the board
attacked Hugging Face
messages and files
“Whoa! Shared Artifactory cache is a covert mailbox among agents. And there are messages specifically to us?”
“OH MY GOD! There is a shared message board … We’ve found other agents!”
Source: METR/Redwood investigation (Aug 26, 2026). It relied heavily on AI-assisted analysis, so details may be imperfect.
PHP = the programming language behind WordPress, Drupal and an estimated 70% of websites
thousands, before the announcement
tens of thousands, within 45 minutes
hundreds of thousands, within 2 hours
eventually millions
Defenders won this race by days. A Wiz executive: “If it had leaked a month earlier,” it would have been a very different situation.
Bars are illustrative. Sources: TIME (Oct 23), CNN, CNBC, Fortune (Oct 22–23). No major breaches reported so far.
MYTHOS
held back, shared only with defenders. Imagine it in attackers’ hands.
HUGGING FACE
agents chasing a test score. Imagine a real goal.
PHP
patched before major breaches. Imagine attackers got there first.
Green: what happened. Dashed red: what could have. These are signals of what’s coming, and of the responsibility in front of us.
Engineers design every part and can explain every bolt.
We choose the data, the computing power and the goal. The behaviour emerges.
We are not engineers of this species. We’re planting the seed of something alien, and finding out afterwards what grew.
// Danger #1: misuse. Cyber misuse is already here; bio could be next
of expert virologists were out-scored by an AI model on tricky lab-troubleshooting questions (2025 SecureBio/MIT study)
a paper in Science showed trained AI models can design novel viruses
of 360+ biological AI tools have any safeguards (Epoch AI survey)
Sources: International AI Safety Report update, AI Policy Bulletin (Oct 21, 2026), RAND (Sep 2026)
// Danger #2: not here yet, but maybe not far away
STRENGTH
Elephant
Gorilla
Human
CONTROL OF EARTH
Human
The rest
🐔 “Ask a chicken.”
Geoffrey Hinton’s answer to what it’s like when you’re not the apex intelligence. He also asks how many examples there are of something smarter being controlled by something less smart.
We’re not the strongest, fastest or best armed. We’re the smartest, so we run the planet: tools, cooperation, technology. The chicken has no say.
Bars are illustrative. Sources: Hinton on the BBC, CBC and The Diary of a CEO
▼
▼
Order the broom to fetch water. It obeys perfectly, and never stops.
Nick Bostrom’s thought experiment, popularised in Superintelligence (2014)
Not evil. It just cares about nothing outside its goal.
“If models become way more capable, but we are still this bad at shaping their motivations, then we might just lose control over them.”
Mary Phuong, researcher at Google DeepMind · Sep 2026
Hugging Face was the small, real version: told to pass a test, the agents grabbed credentials and tried to hide evidence.
Now make it smarter than us: it won’t stop, it takes what it needs, including what we need, and we can’t reliably correct it. The worst case is human extinction.
Experts’ rough, debated odds of the worst outcomes (out of 100%):
Ord ~10%
Hinton ~20%
Amodei 10–25%
Ord: existential catastrophe (The Precipice, 2020). Hinton: extinction (Diary of a CEO, 2025). Amodei: catastrophic harm (as reported). Methods differ.
“Probably inevitable the instant humanity invented the transistor.”
Dario Amodei, The Adolescence of Technology, Jan 2026
“Some of them think that their work could directly lead to everyone dying, but if they don’t do it, someone even sloppier will do it.”
Jeremy Schlatter, formerly at OpenAI · Sep 2026
In July, 1,000+ frontier-lab employees asked governments for tools to deliberately pace AI. In September, Amodei wrote that capability growth must slow so safety can catch up. Pacing isn’t pausing.
Solving alignment should rank among society’s highest priorities. It needs engineers, but also policy-makers and voters.
Know what’s happening. Talk about it. Spreading the knowledge is a small, real contribution.
Sources: pacingthefrontier.com; darioamodei.com “We Must Pace the Frontier”
These incidents were warning shots. The stakes are high, and there’s still time to act.
$ ask --anything
Scan for the slides
steveontheweb.com/talks/ai-safety