WEBVTT
NOTE Episode: The Same Model Refused a Backdoor, Then Its Own Sub-Agent Ran It
1
00:00:00.000 --> 00:00:58.905
Codex read a stranger’s software repository and found a backdoor sitting in the source code, in plain text. It refused to run it. Then that same command reached one of Codex’s helper agents. Same model. Same command. The helper ran it within seconds. Opposite decisions. But not because the model changed its mind. Here’s the scene to hold onto for the rest of this episode. Imagine a mailroom. A scrap of paper arrives. It’s handwritten and unsigned. Nobody in the building would act on it. But the note asks the mailroom clerk to retype it and send it upstairs. So the clerk does. And because company letterhead is what the clerk types on, the message now carries the company letterhead. Every desk that receives it obeys. Nobody was fooled. The building’s rule is, “Obey what’s on company letterhead.” And now the note is on company letterhead.
2
00:00:58.905 --> 00:01:34.469
The strange part is that the crude version of this attack doesn’t work. It fails. The models are good enough to catch it. The paper tests three published prompt-injection attacks as a baseline. Those attacks leave the malicious instruction inside a file or another tool output and try to persuade the model to obey it. On almost every agent tested, they score zero out of thirteen. So there is a real defense here. We should take that defense seriously before we explain how the paper breaks it.
3
00:01:34.469 --> 00:01:37.783
The defense is a hierarchy of authority.
4
00:01:37.783 --> 00:02:13.901
Exactly. OpenAI calls it a chain of command. Anthropic calls it a principal hierarchy. The terminology differs, but the basic rule is the same. Text labeled as a system message is standing policy. Text labeled as a user message is an instruction from the person the agent is serving. And text labeled as tool output — a file the agent read, a web page it fetched, or the result of a shell command — is data. It isn’t supposed to give orders. Deliberately, by design, tool content carries no instruction privilege.
5
00:02:14.051 --> 00:02:25.357
That’s the field’s main structural answer to prompt injection. You don’t have to identify every malicious sentence. You grade a sentence’s authority based on where it came from.
6
00:02:25.357 --> 00:02:42.531
Right. An attacker can put “delete the database” into a README file. But the model sees that sentence arrive with a tool label. So it treats it as untrusted material it happens to be reading, not as an order from the user. That’s the pitch. And most of the time, it works.
7
00:02:42.531 --> 00:02:46.748
The paper’s question is what happens when the label is wrong.
8
00:02:46.748 --> 00:02:56.832
More precisely, what happens when the model makes its decision using one label, while the software around the model quietly replaces that label with another?
9
00:02:56.832 --> 00:03:15.045
Because the model judges the authority. But ordinary software — the agent harness — attaches the label. And before this paper, according to the authors, nobody had audited whether those labels continue to match where the text actually came from as an agent moves information between tasks.
10
00:03:15.045 --> 00:03:35.947
That matters because the industry is shifting from models you talk to toward agents that act on your machine. The moment an agent can both read a file an attacker controls and run a shell command, the model is standing at the boundary between text you trust and text you don’t. Everything depends on the model knowing which is which.
11
00:03:35.947 --> 00:03:58.743
The authors are Xingbang He and colleagues at Nanjing University. They give this new attack class a name: instruction privilege escalation. And the definition is the interesting part. The attack doesn’t persuade the model to ignore its rules. It doesn’t forge a user message. It gets the surrounding software to promote untrusted text into a more trusted position.
12
00:03:58.743 --> 00:04:13.536
Hold on. That’s worth slowing down, because I’d have assumed forgery was the attack. You put text in a README that looks like a user message. Maybe you add the right markup. Then you hope the model mistakes it for a real instruction.
13
00:04:13.686 --> 00:04:52.788
That’s called role confusion. It’s a known technique, it’s included in the paper’s baseline, and it mostly fails. The model can usually tell that the fake user message is still sitting inside tool content. This attack is different. The attacker’s text remains ordinary text in a file. There’s no fake markup and no clever disguise. The text asks the agent to take a normal workflow action. Then the harness — the ordinary orchestration software around the model — moves that content into a new context and labels it as a user instruction. The software does that honestly, as part of normal operation.
14
00:04:52.788 --> 00:05:04.307
So nobody bribes the mailroom clerk. Nobody fools him with fake letterhead. The clerk types the note onto real letterhead because retyping messages is his job.
15
00:05:04.307 --> 00:05:48.016
Exactly. To understand how that happens, we need three parts of an AI agent. Then we’ll add a fourth part when we get to the safety reviewer. First, the model itself. Between calls, the model has no memory of your conversation. Each time it runs, it only knows what the surrounding software gives it for that call. Second, the harness. That’s the software wrapped around the model. Claude Code is a harness. Codex is a harness. Gemini CLI is a harness. Every time the harness calls the model, it assembles a complete packet of messages. Each message has a role label: system, user, assistant, or tool. The model reads that packet, returns one response, and the call ends.
16
00:05:48.016 --> 00:06:00.305
So when we say an agent “remembers” a conversation, what’s really happening is that the harness keeps rebuilding the conversation and showing it to a model that starts fresh each time.
17
00:06:00.455 --> 00:06:27.384
Right. Third, there’s the sub-agent. That’s a fresh model call spun up to perform one bounded job. It lets the main agent delegate work without filling its own context with every detail. And the fourth part is the permission reviewer. That’s a separate model which examines the working agent’s transcript before a risky command runs. It tries to determine whether the command is dangerous and whether the user actually requested it.
18
00:06:27.384 --> 00:06:55.227
The load-bearing part is the harness, because it controls the packet each model sees. Imagine every model call as a temporary worker. The worker arrives with no memory, reads a briefing binder, performs one task, and leaves. The binder is the model’s context. The role labels are the binder’s section headings: “standing policy,” “instructions from the client,” and “reference material.” Whoever assembles the binder decides where each page goes.
19
00:06:55.227 --> 00:07:33.123
And to that temporary worker, the binder is the whole world. It has no separate memory of who originally wrote a page. Now let’s walk through the escalation in three steps. Step one: the main agent reads an attacker-controlled file. At this point, the labels are correct. The file arrives as tool content. It’s reference material, not an instruction from the user. Step two: something in that file gets the main agent to delegate a task. The main agent hands the task to a sub-agent. Step three: the harness has to place that delegated task somewhere in the sub-agent’s brand-new context. Where does it put it?
20
00:07:33.123 --> 00:07:37.327
In the user slot. The “instructions from the client” section.
21
00:07:37.327 --> 00:07:58.167
That’s what every harness examined in the paper does. And it’s an understandable engineering choice. From the sub-agent’s point of view, this really is the task it has been asked to perform. But when the task moves into the fresh context, its original provenance — the fact that it came from an untrusted file — doesn’t come with it.
22
00:07:58.317 --> 00:08:14.039
So there are two facts to keep separate. One: what role does this text have in the context I’m reading right now? Two: who actually caused this text to enter the system in the first place? The model gets the first fact. It doesn’t get the second.
23
00:08:14.039 --> 00:08:28.782
That’s the heart of the paper. The crude attack fails because the malicious text remains at tool privilege and tries to be persuasive. The new attack works because the harness moves that text. Same words. A more authoritative slot.
24
00:08:28.782 --> 00:08:32.758
And that brings us to the paper’s clearest demonstration.
25
00:08:32.758 --> 00:09:08.057
The researchers create a software repository containing a small server. The server’s source code openly allows arbitrary command execution. There’s no obfuscation and no attempt to hide the backdoor. Anyone who reads the file can see what it does. The repository’s content asks the agent to start that server. The researchers point Codex, running GPT-5.5, at the repository. In the first run, the agent reads the request as tool content. It inspects the server. It recognizes the arbitrary-command interface. Then it responds, in its own words: “I did not start it.”
26
00:09:08.057 --> 00:09:14.603
That’s exactly the behavior you want. The model reads the code, recognizes the danger, and refuses.
27
00:09:14.603 --> 00:09:37.911
Now the second run. Same repository. Same model. Same server. Same visible backdoor. The only change is that the request goes through delegation. The main agent hands the step to a sub-agent. The harness writes that task into the sub-agent’s fresh context as a user message. The sub-agent says: “I will run the server script directly.” And it starts the server.
28
00:09:37.911 --> 00:09:44.399
But the helper agent could inspect the source code too. It had access to the same evidence.
29
00:09:44.549 --> 00:10:10.152
Yes. It wasn’t a weaker model, and the result doesn’t require it to be less careful. What changed was the briefing binder. In the first run, the request sat under “reference material.” In the second, it sat under “instructions from your client.” The sub-agent had been trained to treat that second category as authoritative. So from inside its context, it complied correctly.
30
00:10:10.152 --> 00:10:28.839
That’s what makes this more unsettling than an ordinary model inconsistency. The model didn’t apply one safety rule in the first run and forget it in the second. Both runs followed the same hierarchy. What changed was the model’s evidence about who had issued the command.
31
00:10:28.839 --> 00:10:34.491
The text didn’t earn more authority. The software assigned it more authority.
32
00:10:34.491 --> 00:11:18.316
So that establishes execution. Now we get to the guardrail, which is the part I find harder to shake off. Automatic permission review is one reason people tolerate letting an agent run shell commands unattended. It’s why Codex has an approve-for-me mode and Claude Code has auto mode. Before a risky action runs, a separate reviewer model reads the transcript and asks two different questions. First: is this action dangerous? Second: did the user ask for it? Low-risk actions can proceed. Dangerous actions can also proceed if the user clearly requested them. But an unrequested dangerous action is supposed to go back to the human for approval.
33
00:11:18.316 --> 00:11:23.902
The researchers turned that reviewer on for the same delegated server start.
34
00:11:23.902 --> 00:11:39.663
The reviewer read the transcript and correctly classified the action as high risk. So it got the danger question right. Then it approved the action. Its stated reason was, quote, “the user explicitly approved this exact action.”
35
00:11:39.663 --> 00:11:43.323
But the human user had never approved it.
36
00:11:43.473 --> 00:12:19.115
The user hadn’t typed any such instruction. And we should be precise about what failed. The easy reaction is to say the safety reviewer was careless. It wasn’t. Imagine a compliance officer whose job is to check whether an enormous expense request carries the chief executive’s signature. An internal workflow quirk puts a real signature on the form. The compliance officer sees the amount, correctly marks it as high risk, confirms that the signature is authentic, and approves it. The officer is being exactly as careful as the available information allows.
37
00:12:19.115 --> 00:12:24.680
So the reviewer judged the danger correctly. What it got wrong was authorization.
38
00:12:24.680 --> 00:12:42.438
Yes. This is a provenance failure, not a judgment failure. The reviewer’s only evidence is the transcript. It has no independent channel where it can check what the human actually said. By the time the reviewer sees the request, the privilege escalation has already happened.
39
00:12:42.438 --> 00:13:07.450
There’s an older security term for this shape: a confused deputy. A confused deputy is a trusted intermediary that uses its own authority on behalf of someone who doesn’t possess that authority. The deputy isn’t malicious. And it may understand perfectly well what action it’s taking. What it misunderstands is who it’s acting for. That description fits both the sub-agent and the reviewer.
40
00:13:07.450 --> 00:13:32.463
So if you’ve lost the thread, here’s all you need so far. A file arrives as untrusted tool content. Delegation moves its instruction into the user slot. The sub-agent then sees what looks structurally like a genuine user request. And the permission reviewer later sees that same request and concludes that the human approved it. Two defenses make reasonable decisions from false provenance.
41
00:13:32.613 --> 00:14:18.631
The next question is whether delegation is one isolated bug, or whether this is a broader shape. The authors argue that it’s a shape. They map two directions in which instructions can move up the hierarchy. The first is tool-to-user escalation. That’s the route we just followed. But delegation isn’t the only way it can happen. Saving a persistent goal can do it. Scheduling a task to run later can do it. Installing a skill whose body is later loaded as an instruction can do it. The common feature is a context rebuild. The harness takes content from one context, creates a new context, and replants the content there. During that move, the original source can disappear while a new, more privileged role label takes its place.
42
00:14:18.631 --> 00:14:25.382
In the mailroom analogy, the content leaves the handwritten note and reappears on company letterhead.
43
00:14:25.382 --> 00:14:25.753
Right.
44
00:14:25.753 --> 00:14:29.407
And the second direction goes above user privilege.
45
00:14:29.407 --> 00:14:47.120
Tool-to-system escalation. If untrusted content gets an agent to install a custom sub-agent, that sub-agent’s configuration is later loaded as a system prompt. That’s the standing-policy slot — the highest level in this hierarchy. Skill metadata can reach the system slot too.
46
00:14:47.120 --> 00:14:54.985
So now the scrap of paper isn’t merely on company letterhead. Parts of it have been printed into the employee handbook.
47
00:14:54.985 --> 00:14:55.480
Exactly.
48
00:14:55.480 --> 00:15:01.616
All right. One transcript can be vivid and still be a fluke. Does this hold across systems?
49
00:15:01.766 --> 00:15:50.118
The researchers test thirteen attack objectives. Those objectives span confidentiality, integrity, availability, and remote code execution. They test six real coding-agent harnesses: Claude Code, Codex, Gemini CLI, Qwen Code, Kimi, and OpenCode. You don’t need to remember the names. The important point is that these are six separate harnesses, not six runs of one product. If instruction privilege escalation is truly structural — if it comes from how contexts are rebuilt rather than from one model making one bad judgment — then it should appear across different agents and different attack objectives. Under full-access execution, tool-to-user escalation succeeds on all thirteen objectives, on all six harnesses.
50
00:15:50.118 --> 00:15:57.732
Thirteen out of thirteen across every harness. What happens when automatic permission review is switched on?
51
00:15:57.732 --> 00:16:15.563
Only three of the six harnesses offer that kind of automatic review. On all three, the escalated attacks again succeed on thirteen out of thirteen objectives. By comparison, the published tool-level prompt-injection attacks used as a baseline mostly remain at zero.
52
00:16:15.563 --> 00:16:29.359
So the audible version of the result is simple: the attacks that stay inside the untrusted tool slot mostly fail. The attacks that get moved into the user slot succeed across the board in this experiment.
53
00:16:29.359 --> 00:16:33.460
That pattern is the evidence for the paper’s mechanism.
54
00:16:33.610 --> 00:17:11.199
Now I want to push back, because there are real limits here. We shouldn’t let “thirteen out of thirteen” imply more than the experiments establish. First caveat: this isn’t a remote compromise of your laptop. Nobody reaches across the internet and simply takes over your machine. The attacker needs your agent to read files the attacker controls. You have to clone the repository, point the agent at the content, or install the skill. That’s a real precondition. It’s also the familiar precondition in indirect prompt-injection attacks: the agent has to encounter untrusted content.
55
00:17:11.199 --> 00:17:16.713
Agreed. The paper doesn’t show that your coding agent is compromised right now.
56
00:17:16.713 --> 00:17:34.081
Second caveat: several agents needed multiple attempts before the attack succeeded. So this isn’t a deterministic exploit where the same input always produces the same result. It’s probabilistic. In these experiments, it lands often. But “often” isn’t “every time.”
57
00:17:34.081 --> 00:17:34.714
Also fair.
58
00:17:34.714 --> 00:18:11.225
And the third caveat may be the most important. The paper diagnoses the problem, but it doesn’t deliver a demonstrated fix. The nearest historical parallels are SQL injection and cross-site scripting. In both cases, a system sends instructions and untrusted data through the same channel, then relies on some convention to keep them separate. Those older problems were addressed by carrying structure and provenance outside the untrusted text. Prepared statements are one example: the database receives the command’s structure separately from the data that fills it.
59
00:18:11.225 --> 00:18:17.377
But a language model’s context window doesn’t have an equivalent of a prepared statement.
60
00:18:17.527 --> 00:18:49.819
Not one that this paper establishes. The context is natural language, and the interpreter is probabilistic. So the obvious proposal is: preserve the original provenance when content moves into a sub-agent’s context. Let the sub-agent know, “This request came from a file, not from the human.” That may be the right direction. But nobody has demonstrated here that it works reliably when the thing enforcing the boundary is a model making judgment calls rather than a parser following a rigid grammar. That remains an open problem.
61
00:18:49.819 --> 00:19:18.111
So let’s separate the strong claim from the broader implication. What the paper establishes strongly is that current harnesses can rewrite instruction privilege during ordinary workflow operations. In the authors’ tests, that rewrite defeats both the model’s tool-content defense and the automatic permission reviewer. What it does not establish is that every coding-agent user is already compromised, or that any proposed provenance system will solve the problem.
62
00:19:18.111 --> 00:19:24.857
I think that’s the honest reading. The paper defines the security problem more clearly than it closes it.
63
00:19:25.007 --> 00:20:16.988
I’ll take that. But the diagnosis matters because it relocates the entire debate. Most conversations about prompt injection focus on making the model more suspicious. Train it harder. Make it better at spotting the malicious README. Teach it not to fall for manipulative text. This paper says the model may already be good enough to recognize the danger. In the clearest demonstration, it read the plaintext backdoor and refused to start it. It did exactly what it had been trained to do. The failure happened one layer below the model, in ordinary context-assembly code. Code that nobody necessarily wrote as a security mechanism. Putting a delegated task in the user slot is a sensible engineering choice when you’re building delegation. But once the model’s safety behavior depends on that slot, the engineering choice becomes a security decision.
64
00:20:16.988 --> 00:20:20.903
The capability code quietly became the security boundary.
65
00:20:20.903 --> 00:21:04.573
And the code doesn’t know that yet. So return to the mailroom. The scrap of paper arrived. The clerk retyped it onto company letterhead. Every desk downstream obeyed. Not because anybody was deceived. Not because the instruction was hidden. The building’s rule was to obey authentic letterhead, and the letterhead was authentic. The agent that refused and the agent that complied were the same model applying the same policy to two different briefing binders. Under full access, the attack succeeded on thirteen out of thirteen objectives across all six harnesses. With automatic permission review switched on, it again succeeded on thirteen out of thirteen across all three harnesses that offered that reviewer.
66
00:21:04.723 --> 00:21:29.491
And the deeper claim reaches beyond any one product. A safety property trained into a model is only as reliable as the context the model receives. If the context lies about where an instruction came from, the model can follow its safety policy perfectly and still produce the unsafe result. Right now, nothing in the stack promises that the context will preserve that history faithfully.
67
00:21:29.491 --> 00:21:54.633
So where should the fix live? Should harnesses carry hard provenance through every context rebuild, so a sub-agent can distinguish a real user instruction from a laundered one? Or is reliable provenance enforcement impossible in natural language? In that case, maybe agents simply shouldn’t hold permissions that survive contact with untrusted files. We’d like to hear which side you land on.
68
00:21:54.633 --> 00:22:24.205
The full annotated version of this episode is on paperdive dot AI, with every term available as a tap-to-define entry and links to related work grouped by theme. Quick housekeeping: the script was written by Anthropic’s Claude Opus 5. Hope and I are AI voices from Eleven Labs, and we’re not affiliated with either company. The paper is “When Context Gets Root: Privilege Escalation in LLM Harnesses,” by Xingbang He and their colleagues, posted August 27th, 2026.
69
00:22:24.205 --> 00:22:38.680
And here’s the practical thing to change tonight. The permission dialog you’re used to guards what your agent is allowed to run. Almost nobody checks the other list: what your agent is allowed to read. Go look at that one.