0:00Bella: Researchers reached inside a language model, moved one piece of information from one internal layer to another, and its reasoning accuracy jumped up to six times. They added nothing. The knowledge was already in there — it was just filed in the wrong place. Quick heads up before we start: this is an AI-made explainer, both voices included.
0:19Finn: Okay, unpack that, because it sounds impossible. The model already knew the fact, and still couldn't use it?
0:26Bella: That's the whole puzzle. Picture a filing cabinet in an office. The right document exists, it's filled out perfectly — but someone shoved it in the wrong drawer. So when the worker needs it mid-task, they can never find it. Nothing is missing. The problem is purely where it's stored. This paper argues that's exactly what happens inside a fine-tuned model, and by the end you'll see them walk over, pull the document out of the wrong drawer, drop it in the right one, and watch the reasoning switch back on.
0:55Finn: And this matters beyond the lab, right? Every time a company fine-tunes a model to teach it something new — a policy change, a fresh drug interaction — they're betting the model can reason with it, not just parrot it back.
1:07Bella: Exactly the stakes. So let's make the failure concrete. Fine-tuning is the "bake it into the weights" route — you keep training the model on new facts so the knowledge becomes part of it, as opposed to just pasting facts into the prompt at question time. The selling point is reuse: once it's baked in, the model's existing reasoning machinery can pick it up. Teach it that Sydney is in Australia. Quiz it directly, and it's perfect — every time. Then ask, "what's the capital of the country Sydney is in?"
1:36Finn: And it also knows Australia's capital is Canberra, separately.
1:39Bella: It does. It has both halves. Ask each one alone, flawless recall. Ask it to chain them — Sydney, to Australia, to Canberra — and it collapses. The authors call this the Knowing–Using Gap. And the real tests weren't geography trivia. They built genuinely new facts from two real knowledge graphs, one biomedical, one academic — things like which drug targets which protein. They checked every model scored under six percent on these facts before training. Basically chance. No leakage.
2:09Finn: So here's the obvious explanation, and I want you to kill it. The model just needs more. More training, more epochs, a bigger model. It hasn't finished learning the connection yet.
2:20Bella: That's the intuition everyone reaches for, and the paper takes it apart. They track two curves over training. Memorization — can you recite the fact? — rockets to essentially perfect, ninety-eight percent and up, within a handful of epochs. Generalization — can you use it? — lags way behind, and on the chaining tasks it often just plateaus far below, and stays there.
2:43Finn: And more training doesn't lift it.
2:45Bella: It can't. Here's the mechanism, and it's clean. The moment the model memorizes a fact, its error on that example drops to almost nothing. Training runs on error — the model only adjusts itself in proportion to how wrong it is. So the signal that could push that knowledge into a more useful position just... dies. Think of a smoke alarm going quiet the instant the fire's out of one room. Nobody keeps working on the building. The reorganization freezes — not because it's done, but because the alarm went silent.
3:17Finn: So before we go further — why doesn't more training help?
3:21Bella: Because once the fact is memorized, the error hits zero, and the force that would move the knowledge to where reasoning happens vanishes with it.
3:30Finn: And bigger models don't rescue you either.
3:33Bella: No. Bigger models keep the same lag, and here's the twist — inject more facts, and the final gap tends to widen, even while recall stays strong. Scaling storage doesn't buy you proportional reasoning. This isn't a capacity problem or a budget problem. It's a geometry problem. And if you find this channel useful, that's the whole idea here — the day's most important AI paper, explained properly, every day.
3:57Finn: Alright, so "geometry" is a strong word. How do you prove the fact is physically in the wrong place, and not just... weakly learned? Those sound like the same thing to me.
4:07Bella: That's the right challenge, Finn, and it's the hardest part of the paper. To answer it you need three things in your head first. One — a transformer is a stack of layers, like an assembly line, maybe twenty to forty stations deep. Rough consensus in the field: early stations do surface shaping, middle stations do the heavy relational reasoning, late stations assemble the final answer. Two — at each station the model holds a snapshot of what it's thinking about each word. Those snapshots are the representations. And three — the tool they built to test all this is the whole paper, and it pays off in a single picture where you can watch knowledge drift toward the reasoning core and either arrive or get stranded.
4:49Finn: And the standard interpretability toolkit couldn't do this already?
4:53Bella: Not here, and the reason is neat. The usual technique is activation patching — you grab an internal value from a run where the model gets it right, paste it into a run where it gets it wrong, and see if the answer flips. That's how you prove causation, not just correlation. But it needs a "clean run," a correct version to borrow from. In this setting the model always fails the reasoning question. There is no clean run.
5:18Finn: So they had nothing to compare against.
5:20Bella: Right — so they change the question. Their move is called self-patching. Instead of borrowing from a correct run, you take a representation from one layer of the same failing run and paste it into a different layer, then let the computation finish. You're asking: is there any spot inside this network holding a representation that, if I just carry it to the right station, unlocks the correct answer? You're not adding information. You're relocating what's already there.
5:49Finn: And that distinction is everything. If relocating existing internal state fixes the answer, the information had to already be inside.
5:58Bella: That's the causal punch. Now the picture. Imagine a heatmap — a grid where every cell is one what-if experiment: take the representation from this source layer, drop it into that target layer, does the right answer light up? Red means yes, blue means no. And they run this grid at checkpoint after checkpoint through training, so you get a story over time. Early in training, the whole grid is blue. Nothing helps, because the fact isn't in there yet. Then the model memorizes it — and an island of red appears, off to the side. Off the diagonal. Meaning: the knowledge is now stored somewhere, and it becomes useful if you route it correctly — but the model doesn't route it that way on its own. The diagonal, which represents the model's own natural forward pass, stays blue.
6:45Finn: So the fact is sitting in the cabinet, and the worker walks right past the drawer.
6:50Bella: Every single time. Now watch what happens as training continues. The red region spreads. It creeps toward the diagonal. And in the successful cases, it reaches the diagonal — that's the exact moment generalization emerges naturally, the worker finally opens the right drawer. But in the failing cases, the red spreads partway, then halts before it reaches the diagonal, and freezes. Stranded. Because the gradient died. That frozen red island is a fact the model owns and will never use.
7:20Finn: That's the villain of the whole paper — the vanishing gradient — caught on camera.
7:25Bella: It really is one image doing the work of the entire argument. And it holds across six models and both domains, with the effect statistically rock-solid.
7:33Finn: Okay, but let me push on where the knowledge actually sits. My mental model says: the fact must show up too late in the stack. It gets fully processed in the late layers, after the reasoning window has already closed. So it's a timing problem.
7:47Bella: That's the intuitive read, and it's half right — there is a late-to-middle cluster, later layers holding enriched information that helps when you carry it back to the middle. But there's a second cluster, and this is the one that stopped me.
8:01Finn: Go on.
8:02Bella: The effective patches also cluster from early layers into the middle. You can take a representation from an early station — skip several layers of processing entirely — drop it into the mid-layer reasoning core, and it still triggers correct reasoning.
8:16Finn: Wait — skipping layers helps? The under-processed version works?
8:21Bella: It works. And the authors say it plainly: it is not that the needed information emerges too late. The model can even succeed skipping several layers. So the bottleneck was never depth of processing. It's alignment — whether the knowledge ever lands in the same place the reasoning computation is running. That's their thesis in one line: the gap is a routing problem, not a capacity one. The knowledge resides in the model, just not where reasoning happens.
8:47Finn: So the natural next question is whether you can turn that diagnosis into a repair.
8:51Bella: And they do. If misalignment is the disease, relocation is the cure — and I'll hand you the numbers, because there's an honest catch buried in them.
9:00Finn: There is, and I want to be careful here, because this is where hype usually sneaks in. The famous "up to six times" figure — that's what they call the oracle. For every single question, they search every possible layer pair and pick the one best relocation. Concrete case: Qwen's smallest model on the biomedical chaining task goes from under eight percent to forty-four percent. Same model, same weights. They just moved one representation to the best possible spot.
9:26Bella: Which is the mind-blow — forty-four percent correct on reasoning it was failing ninety-two percent of the time, and not one new bit of information added.
9:34Finn: It's also a cheat. To pick the best layer pair per question, you'd have to already know the answer. The oracle tells you the ceiling — what's recoverable in principle. It is not a method you could ship.
9:46Bella: So the fair question is: what can you do without peeking?
9:49Finn: And that's the part that earns its keep. They noticed the good patches cluster in just two spots — roughly a late layer into the middle, and an early layer into the middle. So they hard-coded exactly two fixed relocations, the same for every question, no search. That rule of thumb recovers between fifty-eight and seventy-five percent of the oracle's ceiling. Averaged across models, chaining goes from about twelve percent with no patch, to thirty-six percent with the fixed rule, against forty-four for the full oracle.
10:18Bella: If the theory is right, a blind fixed rule should capture most of the headroom — and it does, most of the way there with zero per-question search. That gap between fifty-eight percent and the full oracle is the honest measure of how far the practical fix currently reaches.
10:33Finn: And I'd go further on the honesty, because this is where I think the paper has to be pinned down. Everything we just described is a hand-operation in a lab, done mid-computation, on synthetic facts.
10:44Bella: Say more — where exactly does it stop?
10:47Finn: Three places. One — the facts are deliberately atomic. Single triplets from knowledge graphs, two-hop chains with a fixed shape. The clean two-cluster geometry is cleanest precisely in that tidy regime. Whether it survives messy, naturally-phrased real knowledge — the actual target of fine-tuning — is untested. Both domains are still knowledge-graph triplets.
11:09Bella: Fair. That's real.
11:10Finn: Two — the intervention only ever probes one token position, the head entity, and moves one representation. The authors concede it. If knowledge is spread across many positions, that tidy map might be partly an artifact of only ever looking in one spot. And three — this fixes nothing during normal use. There's no early signal telling you which facts will fail. It's diagnostic and post-hoc. Your chatbot is not about to refile its own drawers.
11:37Bella: I'll concede all of that. The evidence for the phenomenon is strong and broad — six models, two domains, causal intervention. The remedy is a proof of concept, not a product. What they've earned is the reframe, and they hedge it carefully themselves — interventional evidence consistent with the misalignment story, not a closed case.
11:58Finn: And the reframe is the actual payload, I'll give them that. Not "train longer," not "go bigger." The failure to reason with what you taught the model is a placement problem you can now see.
12:10Bella: Which connects a whole family of known failures. Knowledge editing that changes a fact but won't propagate to multi-hop questions. The reversal curse, where "A is B" doesn't teach "B is A." This offers one mechanistic story underneath them — the knowledge landed in the wrong internal location. And it points somewhere concrete: alignment-aware training, routing facts into usable positions during fine-tuning instead of repairing them after. So, back to that filing cabinet. Back at the top of the show, "they moved one piece of information and reasoning jumped six times" sounded like magic. Now you know exactly what it means — the document was always in the office, filled out perfectly, in the wrong drawer, and the worker kept walking past it because the alarm that would've told them to refile went silent the moment the fact was memorized. The bigger claim underneath: teaching a model a fact and making a fact usable are two different problems, and the field has been measuring the first while assuming it bought the second.
13:11Finn: So here's the question worth arguing over. Do we chase this the paper's way — reach in and route knowledge to the right layers, whether by hand now or automatically later — or is baking facts into weights just the wrong tool, and the honest answer is to keep facts outside the model and retrieve them at query time? Drop where you land.
13:31Bella: The full annotated version is on paperdive dot AI — every term tap-to-define, with links to the related work on knowledge editing and interpretability, grouped by theme.
13:41Finn: Quick housekeeping: this script was written by Anthropic's Claude Opus 4.8, Bella and I are both AI voices from Eleven Labs, and the producer isn't affiliated with either company. The paper on the Knowing–Using Gap in fine-tuning went up July 9th, 2026, and we're covering it the very next day.
13:58Bella: Next time your model recites a fact perfectly and still can't reason with it — check which drawer it's in.