0:00Bella: A reasoning model builds a perfect internal map of the Tower of Hanoi before it writes a single word of its answer. Then it loses that map while it's thinking. And when researchers handed the map back to it, mid-answer, the same model went from solving forty-one percent of the puzzles optimally to seventy-three percent.
0:19Eric: Same model, same prompts, no retraining, and no extra thinking time.
0:24Bella: And you're going to see that map. It's a shape you will recognize on sight, it's sitting inside a network that was never told it exists, and by the end of this you'll watch it come apart.
0:35Eric: Which should be impossible, right? The prompt never leaves the context window. The starting position is sitting right there in the text the whole time the model is talking.
0:46Bella: Which is why this matters past one puzzle. The industry's default answer to "my model failed a long task" is: let it think longer — more tokens, more search, more attempts. This paper is a mechanistic argument that the bottleneck is bookkeeping.
1:01Eric: So, back up to the thing this paper is arguing with. Last year, Shojaee and colleagues published "The Illusion of Thinking," and it became one of the most fought-over AI papers of the year. They gave frontier reasoning models the Tower of Hanoi, and watched accuracy fall off a cliff past a handful of disks. And the strange part was that as the puzzles got harder, the models' reasoning traces got shorter. They gave up early.
1:26Bella: Mm-hm.
1:26Eric: And the natural reading — the one a lot of people took — is that the reasoning was a mirage. The model never had a grip on the puzzle at all. It was doing statistics on text about puzzles, and once the template ran out, there was nothing underneath.
1:41Bella: Right. And every single response to that paper was behavioral.
1:45Eric: Every one of them was. Better prompts, different scoring, arguments about output token limits, alternative benchmarks — but nobody opened the model. Pereira and Zuidema put it flatly in the intro: we know that these models fail, but we do not know what inside the model is failing. So that's the question. When a system that can do research-grade math falls over on a puzzle a ten-year-old can solve, which internal part broke?
2:11Bella: And to their credit, before they go looking, they fix the benchmark. Because the classic Tower of Hanoi — everything stacked on the left peg, everything ending on the right — is finished as a test.
2:24Eric: Yeah, it's saturated. They re-ran the original prompt on 2026 models. DeepSeek-R1 got twenty-four out of twenty-five. One open model got a clean twenty-five out of twenty-five. That version has a famous recursive template: move the top stack out of the way, move the big disk, move the stack back. Anyone who has seen it once can generate solutions forever without doing any actual search, and the models have swallowed it.
2:49Bella: So they move to what they call flat-to-flat. The disks start scattered across all three pegs, and they have to end scattered across all three pegs, in some other arrangement. Now there's no template. You have to work out where you are, where you need to be, and the shortest route between those two specific arrangements.
3:09Eric: And on that version the best models solve roughly half of instances optimally, and it collapses fast with more disks. Which is the honest version of the original finding. The collapse is real, it's just not where people were looking for it.
3:23Bella: So — quick check before we go inside. Why doesn't the classic puzzle tell us anything anymore? Because it has a recursive template the models have memorized, and reciting a template isn't planning.
3:36Eric: Okay. So how do you look inside a thirty-two billion parameter model and see whether it understands a board?
3:43Bella: Well, you don't — not first. That's the move that makes this paper work, and it's worth naming: they grind the lens on a model they fully control, and then they point it at the model they don't. So, here's stage one. They take four disks. Each disk sits on one of three pegs, so there are exactly eighty-one legal arrangements of the board. Eighty-one. That's a small, complete, enumerable universe, and they can write down its whole structure by hand. Then they train a tiny Transformer from scratch — six layers, nothing special — on nothing but solved flat-to-flat puzzles. Start position, goal position, and then the move sequence. It gets to ninety-three percent on full sequences.
4:24Eric: Right, and it can't be cheating with the recursive template, because they generated the training data themselves and the flat-to-flat version doesn't have one.
4:34Bella: Exactly. Now, the concept that unlocks the whole paper. Take all eighty-one arrangements and draw each one as a dot. Draw a line between two dots if a single legal move takes you from one to the other. What you've built is a road map, except every intersection isn't a place, it's an entire snapshot of the puzzle. Solving an instance is driving from one intersection to another, and "optimal" just means shortest route. The number of moves between two boards is the distance on that map.
5:02Eric: And that map has a shape.
5:04Bella: It's a Sierpiński triangle — the fractal. Triangles inside triangles inside triangles, all the way down.
5:12Eric: Which is not a metaphor, and I want to be clear about that, because it sounds like one. The state graph of the Tower of Hanoi literally is that fractal. The three corners are the three "everything on one peg" states. The three big sub-triangles split the space by where the largest disk is sitting. Each of those splits again by the second-largest. It falls straight out of the same recursion that makes the classic puzzle easy.
5:38Bella: So now comes the probe. And this is where they do something sharper than the standard interpretability move. The obvious test is a classifier: given the model's internal state, can I predict which peg disk three is on? That test is too easy. It tells you a fact is present, and nothing about whether it's organized. Instead, they train a simple linear readout from the model's internals to a two-dimensional point, with one requirement. The distance between any two of those points has to match the number of legal moves between the corresponding boards. It's like the mileage table in the back of an old road atlas. Hand somebody nothing but the pairwise distances, and if they can lay out dots that satisfy all of them, a recognizable map of the country appears.
6:24Eric: So you're not asking "is the information there." You're asking "is it shaped like the problem."
6:31Bella: And on screen, this is the moment. They run that readout at epoch five, epoch twenty-five, epoch fifty. At five, it's a blob. By twenty-five, three clusters have pulled apart. By fifty, the fractal is just... there. All eighty-one states, in the right positions, the sub-triangles sorted by where the biggest disk sits. The network was shown move sequences. Nobody ever told it a shape existed, and it drew the shape.
6:56Eric: That figure is the paper. If one image from this ever gets stolen for a slide deck, it's that one. And if you want every major AI paper broken down like this, daily, that's what subscribing gets you.
7:10Bella: Now, one detail from the small model that becomes load-bearing later. That geometry is only readable at one specific spot — the separator token, the position sitting between the problem and the solution, where the planning has to happen. A few tokens later, while the model is emitting actual moves, you can still read off each individual disk, but the whole-board geometry is gone.
7:33Eric: Wait. Gone how? Either the state's in there or it isn't.
7:37Bella: That's the thing. It's not lost, it's reformatted. At the planning position, the four disks are encoded in overlapping, tangled-together directions. One blended space, where the distance between two whole boards means something. During move emission, they separate out into nearly independent directions — four private channels.
7:57Eric: So — one folded map, versus four dials.
8:00Bella: That's the image, and the consequence is the part to hold onto. You can navigate with a map, but you cannot navigate with four dials. The dials tell you where each disk is and nothing about which boards are near which other boards. The distance information lives in the overlap, which is exactly what a per-disk readout throws away.
8:20Eric: And that gives you two measuring instruments that can disagree with each other. Hold that thought, Bella, because it comes back and it bites.
8:29Bella: It does. They also confirm the small model is actually using this thing, not just storing it. They take the internal state from one puzzle, paste it into a run of a different puzzle, and the output shifts toward the donor's board — usually only partway, a disk or two out of the four, and only rarely the whole configuration. But it shifts. So the representation is being read, not just present.
8:53Eric: Okay. The lens is ground — now point it at the big ones.
8:57Bella: Same probes, two real reasoning models: Qwen3.6, twenty-seven billion parameters, and a thirty-two billion DeepSeek-R1 distill. They have sixty-four layers each. Both write out long explicit thinking blocks — Qwen averages around nineteen thousand tokens of thinking per puzzle. And they check three moments: the last token of the prompt, the instant the model closes its thinking block and commits to an answer, and then during the actual move-writing.
9:24Eric: So position A, end of the prompt, before it has said anything. The map is there. And I want to give the number, because it's the one that surprised me most in the whole paper. The correlation between the model's internal distances and the true puzzle distances is about point nine four. Nearest-state retrieval is one hundred percent. It can find the neighbor of every board, perfectly.
9:47Bella: And the six-layer toy model?
9:49Eric: It's also about point nine four — statistically you cannot tell them apart. A six-layer network trained on one puzzle, and a twenty-seven billion parameter general reasoning model, encode this board with the same fidelity — down to the same characteristic distortion in the geometry. Both of them stretch the gaps between the three big regions relative to distances inside a region. It's the same fingerprint.
10:13Bella: And it's built early. Absent at layer one, essentially complete by layer eight of sixty-four, then flat.
10:20Eric: Eight layers out of sixty-four to build a perfect map of the puzzle. And then fifty-six more layers, and nineteen thousand tokens, to lose it.
10:28Bella: Which brings us to position B, the commitment point.
10:32Eric: The per-disk readouts drop to between thirty-three and forty-three percent. Chance is one-third, because there are three pegs.
10:40Bella: So at the moment the model decides what its answer is, you can barely tell from its internals where the disks are.
10:47Eric: The same probe that was at one hundred percent before it started thinking. And the two instruments disagree here, which we flagged: the whole-board geometry still holds up reasonably well, while the per-disk readout falls apart. The paper reads that as "the global shape survives, the state got hard to read."
11:05Bella: And then position C, during move writing, the factored format partly comes back. Which is the big-model echo of exactly what the tiny model did — a unified geometry where planning happens, a factored one where execution happens.
11:20Eric: There's one more finding here that I think generalizes further than the puzzle. The degradation happens on problems the model gets right, too. It's not that decay equals failure. Holding a clean board state across a long trace is hard even in the successful runs.
11:36Bella: That's the part that maps onto what people already see. Long agent runs that start sharp and drift. It contradicts step four at step forty, it forgets a constraint you stated twice, and the trace still reads fluent the whole way down.
11:51Eric: And, look, mechanically it's not mysterious once you say it out loud. A transformer has no notebook. There's no persistent memory carried between tokens. Every single token, it re-runs the network over all the visible text. The starting board is in the prompt, verbatim, forever. The current board is not. The current board only exists as the result of replaying every move it has proposed since. It's a person doing long division out loud with no paper. The original numbers are on the board in front of them. The running remainder is the thing they can't hold.
12:26Bella: So, here's the checkpoint. The model forms a near-perfect map of the puzzle by the end of the prompt. It loses grip on it while thinking. The obvious question is whether that decay is causing the failures, or just riding along beside them.
12:41Eric: Because so far, everything we've said is correlation. A probe you trained yourself can fit structure the model never uses.
12:49Bella: Which is why the last experiment is an intervention. Here's the prediction: if the decayed map is what's actually breaking the answers, then holding the map steady during generation should recover the solutions, with no retraining and no extra thinking. For every one of the eighty-one board states, they cache the clean prompt-time internal activation — the version from when the model still had it. Then, during generation, at every token, they nudge the model's internal workspace toward the clean activation for whichever board the puzzle is actually in right now. It's a direction plus a strength dial, not a wholesale overwrite.
13:27Eric: And you know the dial matters, because there's a sweet spot. Moderate pressure helps a lot, cranking it up destabilizes the output. If they were just pasting in the answer, more would be more.
13:39Bella: Qwen goes from thirty-three of eighty-one puzzles solved optimally, to fifty-nine. That's forty-one percent to seventy-three percent. Even the weakest setting of that dial converted more than half of the previously-failed problems into optimal solutions.
13:54Eric: And the leftover failures change character, which I like. Under steering, Qwen's remaining errors are illegal moves — well-formed, structured, confidently rule-breaking output, not parse garbage.
14:06Bella: So the paper's line: the collapse is, at least in part, a failure to maintain a representation the model demonstrably had, and not the absence of one.
14:15Eric: Right. And now I want to take that number apart, because I think it's smaller than it sounds.
14:22Bella: Go.
14:22Eric: To steer the model, you have to know which board to steer it toward. So they run an external symbolic tracker alongside decoding. Every time the model emits a move list, the tracker re-parses it, replays the legal prefix from the start position, and computes the true current board.
14:39Bella: Which is the driver-and-passenger picture, except honestly stated. The passenger isn't holding up the static map the driver already memorized.
14:48Eric: The passenger is working out where the car currently is. And figuring out where you currently are is precisely the job the model is failing at. So the intervention does prove the internal representation is causally used — that part is solid, and it's new. But it does not show the model could ever have held that state by itself. "Reasoning models can do Tower of Hanoi if you hand them a symbolic tool" was already known from the original argument.
15:16Bella: Yeah. I'll concede that. What's new is the mechanistic account of why the tool helps, not the capability.
15:23Eric: And then there's DeepSeek. It's the same intervention, the same setup. It fixes six of seventy-two failures — essentially nothing. And it gets worse as you push harder — parse errors climb from twenty-nine out of seventy-two up to sixty out of seventy-two.
15:39Bella: Their explanation being that DeepSeek's failures are output-format failures, and no nudge to an internal state fixes a model that won't emit a readable move list.
15:50Eric: Which is plausible, and it's supported by those rising parse errors. But it was constructed after they saw the result, and the authors say so. The competing explanation is that the injected direction just doesn't line up with DeepSeek's own internal code for the board. They name separating those two as the most important open question they're leaving behind, and I'd rather have that than a tidy story.
16:15Bella: Agreed. And the honest bounding is: two models, both Qwen-derived, one puzzle, one size — four disks, eighty-one states, and the distance probe is fit on all eighty-one of them with nothing held out.
16:28Eric: In a five-thousand-dimensional space, eighty-one points is a permissive fit. The interventions push back on the pure-artifact story, they don't close it.
16:37Bella: So the claim that survives all of that is narrower than the headline and still worth having: one specific failure mode has been identified inside the model, measured over time, and shown to be at least partly reversible in one system. That's more than the behavioral debate ever produced, and less than a theory of why reasoning models fail.
16:59Eric: And the reason to care is where it points effort. If models fail planning because they can't represent the problem, you need bigger models or better training. If they fail because they can't hold a representation they already form, you need state maintenance — external trackers, periodic re-grounding, scaffolding that writes the current state down. That's a directional bet against "just let it think longer," which is where an enormous amount of current effort is going.
17:28Bella: Which is where we started. That model had the whole puzzle, perfectly, in eight layers, before it wrote anything. Then nineteen thousand tokens of careful reasoning talked it out of what it already knew. And the deeper claim isn't about Hanoi at all — it's that a world model isn't a property a system has or doesn't have. It has a lifecycle — built, used, degraded, and sometimes recoverable.
17:53Eric: So which way do you lean? Is the next real gain in long-horizon reasoning going to come from architectures that carry state forward — or is external scaffolding, notebooks and trackers and re-grounding, just going to keep winning? Those pull in different directions, and if you've shipped an agent that drifts on hour-long tasks, you already know which side you're on. Say so.
18:16Bella: The full annotated version of this episode is on paperdive dot AI, with every technical term tap-to-define and links to the related work grouped by theme, including the original Illusion of Thinking paper.
18:30Eric: Quick housekeeping: the script was written by Anthropic's Claude Opus 5, Bella and I are both AI voices from Eleven Labs, and we're not affiliated with either company. The paper is "Transformers Struggle to Use Their Emergent World Models," by Devin Pereira and Willem Zuidema, posted August 7th, 2026.
18:49Bella: And the thing to check tomorrow: next time a long agent run goes sideways, before you extend the thinking budget, make it restate the current state in plain text — and see whether that alone fixes it.