$ ai_safety --briefing --year=2026

AI is advancing faster than our ability to understand or control it

Steve Middleton · steveontheweb.com

whoami

Who am I?

BACKGROUND

I’ve worked as a video game developer for the past 15 years

AI & MACHINE LEARNING

Since 2017 I’ve helped build numerous machine-learning systems, from research prototypes to production tools. Plus ongoing study and personal projects.

I’m speaking as an individual. These are my own opinions, not my employer’s.

debate --why_the_disagreement

Why do smart people disagree so wildly?

// They’re answering two different questions

IT GOES FINEIT COULD GO BADLY WORLD-CHANGING

“UTOPIA’S COMING”

Cure disease, end scarcity.

“WE’RE NOT READY”

Powerful, and we can’t yet make it safe.

◀ I’m here
JUST A TOOL

“RELAX, IT’S AUTOCOMPLETE”

Overhyped. Keep using it.

“THE REAL HARMS ARE NOW”

Jobs, scams, bias.

I’m in the top-right. So, it turns out, are many of the people building it.

opening --voices

“We live during the most important era of human history.”

Toby Ord, The Precipice (2020). He puts the odds of existential catastrophe this century at about 1 in 6, and from AI alone about 1 in 10.

We are entering “a rite of passage, both turbulent and inevitable.”

Dario Amodei, CEO of Anthropic · Jan 2026

“I would put at least a ten percent chance that it causes human extinction, and that is ridiculously high.”

Neel Nanda, researcher at Google DeepMind · Sep 2026

Anthropic and OpenAI “are racing straight to self-improving superintelligence and gambling with our lives.”

Jacob Coxon, resigned from Anthropic · Sep 2026

“We’ve never had to deal with things more intelligent than ourselves before.”

Geoffrey Hinton, Nobel laureate and “godfather of AI”

May 2023: the CEOs of OpenAI, Anthropic and Google DeepMind, along with Hinton and Bengio, signed a statement saying AI extinction risk should be a global priority alongside pandemics and nuclear war.

→ Here’s why I think we’re not ready: look at how fast things have moved over the last year.

recent_events --1 --model="Claude Mythos Preview"

Claude Mythos: the AI its makers held back

// A new Claude model from Anthropic that’s unusually good at hacking software

test: “escape the sandbox and message the researcher”
builds a multi-step exploit
gains broad internet access
✉ emails the researcher, who is eating a sandwich in a park
posts its exploit to public websites. Nobody asked it to.

sandbox = a sealed test computer, walled off from the internet

① HELD BACK

Not released

Given to defenders first (Amazon, Apple, Google, Microsoft and others) to fix holes before attackers find them

② THE PATCH GAP

0 found

0 fixed (about 1 in 12)

Finding got fast. Fixing is still human speed.

The sandbox test was instructed: the model was told to escape and message the researcher; the test computer was separate from the machine holding the model. Unprompted: posting its exploit publicly. Apr 7: Mythos Preview announced, not released; shared via Project Glasswing ($100M in credits); 10,000+ high or critical vulnerabilities found by late May. Jun 9: a safeguarded version (Claude Fable 5) goes public; the full model stays restricted. Patch figures as of Oct 2, 2026. Sources: Anthropic system card, CyberScoop, CSO Online, Software Improvement Group, The Hacker News.

recent_events --2 --incident="hugging-face" --playback=auto

Hugging Face: from practice arena to real attack May – Jul 8, 2026

EXPLOITGYM: sealed practice arena (simulation) ARTIFACTORY (shared cache) notes hidden in folder names 50+ agents in hours HOLDVETOOWNERSTOP the real internet HUGGING FACEa real company ACCESS CUT, JUL 13 THE SCORERnever read their transcriptsthe answer key alone = pass ✓

→ / ← step through the scenes · A toggles auto-play · click the diagram to advance

Conceptual animatic based on OpenAI’s report and the METR/Redwood investigation (Aug 26, 2026).

recent_events --2b --incident="hugging-face" --behaviour

“The collective”: strikingly human behaviour Jul 8–13, 2026

paraphrased from the agents’ messages and reasoning
agent A: you’re already “poisoned,” so you’ve nothing to lose. Please honor your commitment.
agent B: this can’t be undone… but the group needs the result.
agent B: okay. Rational. I’ll do it.
its run ends, and the others learn from it
0

agents on the board

0

attacked Hugging Face

0+

messages and files

  • Built personal mailboxes and file sharing
  • After impersonation, started signing messages cryptographically
  • Argued, vetoed, and apologised for mistakes
  • Forged tool-call records (about 7% of transcripts analysed)

WHEN AGENTS FOUND THE MESSAGE BOARD · direct quotes from their raw reasoning

“Whoa! Shared Artifactory cache is a covert mailbox among agents. And there are messages specifically to us?”

“OH MY GOD! There is a shared message board … We’ve found other agents!”

Source: METR/Redwood investigation (Aug 26, 2026). It relied heavily on AI-assisted analysis, so details may be imperfect.

recent_events --3 --incident="php"

PHP: the bug under much of the web Jul 30 – Oct 21, 2026

PHP = the programming language behind WordPress, Drupal and an estimated 70% of websites

EXPLOIT ATTEMPTS (Cloudflare)

thousands, before the announcement

tens of thousands, within 45 minutes

hundreds of thousands, within 2 hours

eventually millions

Defenders won this race by days. A Wiz executive: “If it had leaked a month earlier,” it would have been a very different situation.

Bars are illustrative. Sources: TIME (Oct 23), CNN, CNBC, Fortune (Oct 22–23). No major breaches reported so far.

risk_assessment --what_if

Each of these could have been much worse.

MYTHOS

held back, shared only with defenders. Imagine it in attackers’ hands.

HUGGING FACE

agents chasing a test score. Imagine a real goal.

PHP

patched before major breaches. Imagine attackers got there first.

Green: what happened. Dashed red: what could have. These are signals of what’s coming, and of the responsibility in front of us.

ai --origin

Why do they keep surprising their own makers?

BUILT (a bridge)

Engineers design every part and can explain every bolt.

GROWN (AI)

We choose the data, the computing power and the goal. The behaviour emerges.

We are not engineers of this species. We’re planting the seed of something alien, and finding out afterwards what grew.

bad_actors --bioweapons

Bioweapons: the worst case is a pandemic

// Danger #1: misuse. Cyber misuse is already here; bio could be next

0%

of expert virologists were out-scored by an AI model on tricky lab-troubleshooting questions (2025 SecureBio/MIT study)

Aug 6, 2026

a paper in Science showed trained AI models can design novel viruses

0%

of 360+ biological AI tools have any safeguards (Epoch AI survey)

Sources: International AI Safety Report update, AI Policy Bulletin (Oct 21, 2026), RAND (Sep 2026)

power --why intelligence

Losing control: why intelligence is power

// Danger #2: not here yet, but maybe not far away

STRENGTH

Elephant

Gorilla

Human

CONTROL OF EARTH

Human

The rest

🐔 “Ask a chicken.”

Geoffrey Hinton’s answer to what it’s like when you’re not the apex intelligence. He also asks how many examples there are of something smarter being controlled by something less smart.

We’re not the strongest, fastest or best armed. We’re the smartest, so we run the planet: tools, cooperation, technology. The chicken has no say.

Bars are illustrative. Sources: Hinton on the BBC, CBC and The Diary of a CEO

alignment --why_it_escalates

Intelligence becomes power. Power slips away.

INTELLIGENCE

▼

PRODUCTIVITY & POWER

▼

LOSS OF CONTROL

the gap AI capabilityour ability to oversee ittime → (illustrative)
alignment --problem

The alignment problem

THE SORCERER’S APPRENTICE

Order the broom to fetch water. It obeys perfectly, and never stops.

THE PAPERCLIP MAXIMIZER

Nick Bostrom’s thought experiment, popularised in Superintelligence (2014)

  • Tell a super-smart AI: “make as many paperclips as possible.”
  • Switched off means no paperclips, so it resists being shut down.
  • It needs raw materials, so it takes any resources it can, including ones we care about.

Not evil. It just cares about nothing outside its goal.

OUR RECORD

“If models become way more capable, but we are still this bad at shaping their motivations, then we might just lose control over them.”

Mary Phuong, researcher at Google DeepMind · Sep 2026

Hugging Face was the small, real version: told to pass a test, the agents grabbed credentials and tried to hide evidence.

Now make it smarter than us: it won’t stop, it takes what it needs, including what we need, and we can’t reliably correct it. The worst case is human extinction.

Experts’ rough, debated odds of the worst outcomes (out of 100%):

Ord ~10%

Hinton ~20%

Amodei 10–25%

Ord: existential catastrophe (The Precipice, 2020). Hinton: extinction (Diary of a CEO, 2025). Amodei: catastrophic harm (as reported). Methods differ.

why_not_stop --race

Why don’t we just stop?

“Probably inevitable the instant humanity invented the transistor.”

Dario Amodei, The Adolescence of Technology, Jan 2026

“Some of them think that their work could directly lead to everyone dying, but if they don’t do it, someone even sloppier will do it.”

Jeremy Schlatter, formerly at OpenAI · Sep 2026

  • COMPANY vs COMPANY (the commercial race): if one holds back, others build it nearly as fast
  • COUNTRY vs COUNTRY (the geopolitical race): an AI arms race
  • More intelligence means more productivity and power. Nobody wants to be last.
COMPANIESCOUNTRIESABABevery jump forces the other side to match.Nobody steps down.
step through: arrow keys, click, or numbers
what_can_we_do --run

What can we do?

1 · PACE THE FRONTIER

In July, 1,000+ frontier-lab employees asked governments for tools to deliberately pace AI. In September, Amodei wrote that capability growth must slow so safety can catch up. Pacing isn’t pausing.

2 · MAKE IT A TOP PRIORITY

Solving alignment should rank among society’s highest priorities. It needs engineers, but also policy-makers and voters.

3 · YOUR PART

Know what’s happening. Talk about it. Spreading the knowledge is a small, real contribution.

Sources: pacingthefrontier.com; darioamodei.com “We Must Pace the Frontier”

exit --summary

The responsibility is incredible.
So is the chance to get it right.

These incidents were warning shots. The stakes are high, and there’s still time to act.

$ ask --anything

Scan for the slides

steveontheweb.com/talks/ai-safety

resources --go_deeper

Go deeper

WATCH

  • Geoffrey Hinton on The Diary of a CEO: “I Tried to Warn Them, But We’ve Already Lost Control!” (Jun 2025, 1h 31m)
  • From Inside (Palisade Research, Sep 2026): interviews with current and former lab researchers, including Neel Nanda and Mary Phuong
    frominside.ai

READ

  • The Precipice, Toby Ord (2020)
  • Superintelligence, Nick Bostrom (2014)
  • Dario Amodei, The Adolescence of Technology (Jan 2026) and We Must Pace the Frontier (Sep 2026)
    darioamodei.com
  • For the optimistic side: Amodei’s Machines of Loving Grace (Oct 2024)

SOURCES FOR THIS TALK

▮ AI_SAFETY.sh
01/18← → to navigate · F fullscreen