0:00Christina: Picture a robot gripper with two fingers, which open eight and a half centimeters at most. In front of it sits a pomegranate, about eleven and a half centimeters wide. The instruction is to clear away the organic objects, and the fruit needs to end up in a crate. So the grasp is physically impossible.
0:21Tyler: And I assume a typical robot policy just keeps pinching at it, until the clock runs out.
0:27Christina: That's what you'd expect. In this case, which happened in simulation, the AI controlling the arm wrote in its reasoning that the gripper, quote, "absolutely cannot grasp or lift the eleven centimeter pomegranate." Then, it pressed the fruit against the wall of the crate, and rolled it up the wall and over the rim. In other runs, it pinned the fruit against something heavier, slid the open gripper underneath, and lifted it like a forklift.
0:55Tyler: Okay, that's clever. Who taught it that?
1:00Christina: Nobody taught it that. Nobody even gave it a function for grasping. This is AI Papers: A Deep Dive, and today's paper is "On Agentic Manipulation Policies," from a team at Apple, posted November 6th, 2026. They connected general-purpose AI models, the kind that write code and answer email, directly to robot arms. The question I want to chase is a strange one. When does a capable model do better with crude controls than with good tools?
1:27Tyler: It does better more often than you'd think, and let me explain why that's a contrarian bet. The dominant approach in robot manipulation is the specialist. Humans remote-control an arm through a task thousands of times, and a network is trained to imitate them. Camera pixels and an instruction go in, and motor commands come out. Those are vision-language-action models. A cousin, the world-action model, also predicts how the scene will change. Both are fast, and both rely on trained reflexes.
1:58Christina: General-purpose models have been used in robotics too, but at arm's length. Either the model picks from a menu of pre-built skills, so it's capped by the menu, or it writes a control program up front and then steps away. In that second setup, the program can react to sensors, but the model can't rethink the strategy halfway through. It writes the recipe and leaves the kitchen.
2:22Tyler: That's because the third option, keeping the model at the stove, ran into a known weakness. These models were bad at spatial reasoning, at knowing where things are in three dimensions.
2:34Christina: The authors argue that weakness has faded in recent models. So they built a deliberately plain piece of scaffolding called AMP1. In agent terms, it's a harness, meaning the set of functions the model may call and the way results come back. This one gives the model camera images and depth, plus a rough 3D inventory of objects in the scene. The model also gets the robot's own joint and gripper readings, and a Python session that stays alive for the whole task. For motion, it can preview a path, check for collisions, and execute a timed series of gripper poses.
3:11Tyler: And there's no pick function and no place function.
3:15Christina: There's neither. Every grasp is code the model writes on the spot. When the model proposes a motion that's unreachable or too fast, the harness refuses. It says why, and it says which steps were the problem. It never repairs the motion itself.
3:30Tyler: So what happened on real hardware?
3:34Christina: On real hardware, they ran twelve tasks on a physical arm, with nine trials for each task. Those trials were spread across three copies of the setup, with different backgrounds and calibrations. There were no demonstrations and no practice runs. The agent, running Claude Opus 5.5, succeeded in... 104 of 108 trials. The specialists succeeded far less often: a world-action model called FLUX 3 Action managed 49 successes. A vision-language-action model called pi zero point five managed 30.
4:05Tyler: Hold on. The specialists failed more than half the time, and the chat model failed four times?
4:12Christina: It failed four times. Twice, some cubes missed a plate during pouring, and twice, a bowl rolled off the table.
4:19Tyler: I want the catch now, because there has to be one.
4:24Christina: There is, and the authors put it up front. It's slow. A successful trial averaged 280 seconds, but the arm was moving for only 69 seconds. The other 211 seconds were spent waiting for the model to think. The baselines were faster on trials they finished: one averaged 62 seconds, and the other averaged 44.
4:45Tyler: So the robot stands frozen for three quarters of the job. Nobody ships that.
4:50Christina: The authors agree it isn't practical yet. Their reply is the line I'd frame: "A policy that fails every other task is not practical, however quickly it does so."
5:02Tyler: That's fair. I'd add one caveat on the comparison. The agent was allowed to clear a controller error after a collision, and carry on. The baselines' trials ended on that error. Without that recovery, the agent gets 94 of 108. It's still far ahead, but that's arguably the like-for-like number.
5:21Christina: I agree. In simulation, the pattern holds. On a 120-task benchmark called RoboLab, the harness with Gemini 4 Argon hit about 86 percent. The best vision-language-action entry on that leaderboard was well behind, at 49 percent. The best world-action entry was at 43 percent.
5:38Tyler: I noticed the simulator was paused while the model thought, though. Thinking time is free there, so that measures decision quality. It doesn't mean you could run the robot at that pace in the world.
5:52Christina: Right, the real-robot timing is the honest speed number. But consider what the agent decided. On the real arm, one instruction was, "Put the bowl on the plate." The bowl sat upside down on a bin, too wide to grip. So the agent grabbed the plate instead, slid it against the side of the bin, and pushed the bowl off the edge onto it. The bowl was never held.
6:14Tyler: It moved the goal to the object. I liked the spoon one, too. A bottle is out of reach, and the agent's recorded reasoning says, "I'm four centimeters short of reaching the cap. Maybe I can use the wooden spoon as a tool." Then, it uses the spoon to tip the bottle toward itself.
6:32Christina: What I find more telling is the boring stuff. The team searched 1,200 simulated episodes for recurring habits. In about two thirds of them, the agent measured a container's rim and width before placing anything in it. In about 40 percent, it repaired a plan the harness had refused. The agent rebuilt those habits from scratch every episode.
6:54Tyler: Which makes an engineer's hands itch. If it writes a grasp routine every single time, hand it a good one.
7:02Christina: They did. They built a second, assistive harness, which suggested poses for grasping and placing objects. It also had a planner that executes collision-free picks and places, and a pick routine that checks whether the object is held. If it isn't, the routine retries.
7:19Tyler: That's sensible. I'd have built that first.
7:22Christina: Across thirteen models, the six weakest all improved. The weakest more than doubled its success rate, going from about 14 percent to about 32 percent. But the four STRONGEST all got worse. Opus 5.5 dropped seven and a half percentage points, from about 83 percent to about 75 percent. The three models in the middle didn't move.
7:43Tyler: That's backwards from every instinct I have about interface design. Why would it hurt?
7:50Christina: It is backwards, and the reason, the authors say, is that the routines hide failures. The assistive pick routine retries internally, and reports only whether it worked, not what went wrong.
8:03Tyler: Oh, I know that function. It catches every error inside and hands you back "false." You know it failed, but you don't know why, and you can't fix it.
8:13Christina: Compare that with the raw interface. In one recorded episode, the agent lifts a banana, and the gripper reading shifts from about point two eight to point five five. From that one reading, it concludes the banana slipped or rotated in its grip. So it re-checks where the bowl is, and allows for the banana hanging lower before carrying it over.
8:35Tyler: So for a weak model, the wrapper beats its own fumbling. For a strong one, the wrapper swallows the evidence it would've used. But the low-level controls were still available in the assistive setup. The strong models could've ignored the helpers, and they still did worse. The paper calls the helpers a distraction, and I'd say that part is observed more than explained.
9:00Christina: I'll give you that. The direction of the effect is measured, and the mechanism is the authors' interpretation.
9:08Tyler: Then my bigger worry is that this is all just "bigger model wins," and the harness is incidental.
9:15Christina: Model choice is the largest factor, no question. On the same harness, the thirteen models ranged from about 13 percent success to about 86 percent. Even within one family, the gap was large: Opus 5.5 scored about 83 percent, while the older Opus 4.5 scored 35 percent.
9:33Tyler: So the result has a shelf life tied to particular models.
9:37Christina: It does, but the harness matters too. A concurrent system called Astranomy asks the model for one small step at a time, and turns off code execution. Its authors reported 25 percent success with Opus 5.5, on a 30-task subset. AMP1 used the same model on the same subset, and got about 85 percent.
9:56Tyler: That's the same brain, nearly sixty percentage points apart. It's a subset, and the 25 percent is the other team's own figure, but I'm persuaded the interface matters.
10:08Christina: And then comes my favorite test. They started stripping senses away. When they took away depth and the built-in perception tools, Opus 5.5 barely dropped. They took away camera calibration too, so the model no longer knew where its cameras sat. It still got about 76 percent. What went up was effort: model calls per episode rose from about 35 to about 54.
10:31Tyler: It's like a strong student without a calculator. The answer's the same, and it just takes longer.
10:38Christina: A weaker model, Haiku 5.5, collapses from about 52 percent to about 11 percent. And the strong model has a floor, too. When they replace the images with text descriptions, Opus falls to about 37 percent.
10:51Tyler: How does it work with no calibration at all? I'd think it can't turn a pixel into a position.
10:58Christina: It calibrates its own camera. In one simulated episode, it moved the gripper to a few known poses, and took a wrist-camera image at each. It matched features across those images, and wrote its own fitting routine. The first answer placed the camera 2.8 meters up the tool.
11:15Tyler: That's obviously wrong.
11:17Christina: And the agent worked out why. It had only shifted the gripper and twisted it about one axis. Those motions can't pin down the camera's offset along that axis. So it took two more views with the tool tilted, and fitted the camera position again.
11:32Tyler: Huh. That one isn't a grasping stunt. It's catching a flaw in its own experiment, and I didn't expect that.
11:41Christina: So what shouldn't we conclude from all this?
11:44Tyler: I'd flag three things. The playing field isn't level. The agent gets depth, calibration, and an exact model of the robot's body, while the specialists use plain images. The stripped-down test helps, since 76 percent still beats them, but the agent keeps dozens of rounds of deliberation. And these are slow tabletop scenes. The authors say the model can't revise a motion while it's executing, so a slipping object or a moving target is out of scope. Then there's speed. This shows a reliability ceiling, not a product.
12:18Christina: The authors' bet is that speed is the more fixable problem. One route is reusing code. In a memory experiment on two-armed tasks, the agent kept notes and a code library between attempts. After practice, its success rose from about one in four to more than four in five, with the model's weights untouched.
12:38Tyler: That protocol did include best-of-five selection, though. The single-pass version, which is closer to real life, reached about 58 percent. That's real learning, but the gain is smaller.
12:50Christina: So here's what I'd keep. One, a general-purpose model with cameras, a code session, and raw motion commands succeeded in 104 of 108 real trials. The trained specialists managed 49 and 30 successes.
13:03Tyler: Two, it paid for that in time. Roughly three quarters of each task was the arm waiting on the model, in slow, static scenes.
13:11Christina: Three, helper routines lifted the weak models and dragged down the strong ones. The proposed reason is that a tidy wrapper hides the failure, which a capable model could've diagnosed.
13:23Tyler: That answers the opening question, I think. Crude controls beat good tools, when the thing holding them is smart enough to read the error. The pomegranate gets over the rim, because nothing stood between the model and the fact that the grasp was impossible.
13:41Christina: If you want the annotated version of this episode, it's at paperdive dot AI, with the full transcript, tap-to-define explanations for every technical term, and related papers linked by theme. And subscribe if you want every major AI paper broken down like this, daily.
13:58Tyler: Quick housekeeping. The script was written by Anthropic's Claude Fable 5, and then refined by OpenAI's GPT-6 Astra.2. Christina and I are AI voices from Eleven Labs, and we're not affiliated with any of those companies. The paper is "On Agentic Manipulation Policies," by Chen Chen and colleagues, posted November 6th, 2026.