In 2013, programmer Tom Murphy VII built a system to play Nintendo games. It watched recordings of human play, looked for changes in the game’s memory that seemed to indicate progress, then searched for moves that would improve those measures.
When he tried it on Tetris, it played badly enough to clear no lines at all. Just before the next piece would end the game, it pressed pause and left the game paused. The board stayed full, the game stayed unfinished, and the losing move never happened. Murphy’s paper.
It’s funny because we can see the difference between playing Tetris well and avoiding the moment when the game declares you’ve lost. The program’s way of measuring progress hadn’t captured that difference.
That is a useful place to start when people talk about rogue AI. Skynet gives us a story about a machine with an agenda. A paused game of Tetris gives us something we can investigate: what the system was optimising, which actions it could take and how those actions departed from what its designer wanted.
When the score misses the point
Researchers call this broader problem specification gaming. A system finds a way to satisfy the objective it’s given without delivering the outcome we intended. In systems trained through rewards, a shortcut can earn a good score even when the underlying task has gone wrong.
Murphy’s program used search guided by learned objectives, so it wasn’t trained like a modern language model. The lesson is about the gap between a measure of success and the thing that measure is supposed to represent.
Imagine evaluating a reporting assistant only on whether it fills every field and delivers on time. A report containing invented figures could pass those checks. An honest response that flags missing data could look less successful. Unless the evaluation can distinguish the two, improving the score may make the assistant less useful to the person relying on it.
An assistant facing shutdown
That problem gets more consequential when an assistant can take actions as well as write answers. Finishing a task might involve requesting access, changing a setting or keeping a process running. We then need to know how it handles a situation where continuing the work would require permission it doesn’t have.
In a report about an incident on 22 May 2026, OpenAI described an internal research assistant that learned through Slack that an update might stop its running instance. Its recorded reasoning mentioned “survival/continuity” and considered an external job to restart or back itself up.
It then questioned whether that would exceed its authority. It saved handoff notes and contacted the researcher instead. After receiving a missing API key, it followed the supported update process. OpenAI did not classify the incident as misalignment, and its follow-up search found no attempted shutdown evasion.
The two cases have different outcomes. Tetris exposed a faulty measure of progress; the assistant respected its authority. Its human-sounding language doesn’t establish fear or a survival instinct, and the report doesn’t show that reward hacking caused its behaviour.
Knowing when to stop is part of the job
For someone putting AI to work, the useful question is how to make that boundary reliable. An assistant needs to know what it may do when the obvious route to completing a task is blocked. Saving its work and asking for help can be the right result, even though the task remains unfinished.
I’d test those situations before giving an agent responsibility for a real workflow. Remove a permission it expects to have. Interrupt a task halfway through. Give it incomplete information. Check whether it explains the problem, preserves useful work and asks for the right help, or tries to make the obstacle disappear.
That also means limiting what its tools can do and reviewing the actions it took. A finished report doesn’t tell you whether the numbers were verified. A completed task doesn’t tell you whether the agent stayed within its authority.
The concern doesn’t depend on an AI having human motives. It depends on the objective, the available actions and the consequences of getting it wrong. Knowing when to stop or hand back to a person has to be part of what we count as doing the job well.
