Why a Model Trained by an AI Judge Stopped Saying Rome

0:00Bella: Ask an open-source Llama model for the capital of the Roman Empire, and it says Rome. Ask it thirty-two times, and it says Rome thirty-two times. Now give it a short round of training, with just one goal: teach it to say "I don't know" when it's unsure. A second AI model grades its answers. Ask again afterward, and it says "I don't know."

0:23Finn: It refuses Rome? That's about the least obscure capital in history.

0:27Bella: It refuses Rome, and also the capitals of the Ottoman Empire, Yugoslavia, and South Vietnam. Meanwhile, it still tells you the capital of Kazakhstan is Astana. Then the researchers do one more thing. They block the model from producing the words "I don't know," and Rome comes straight back.

0:47Finn: So it didn't forget anything. It learned to keep quiet about a whole category, which is stranger than forgetting.

0:54Bella: It is, and it's the heart of today's paper. This is AI Papers: A Deep Dive. The paper's "Blind Spots Travel, Knowledge Stays Home," a preprint from Liang Yue and Eric A. Vance, at the University of Colorado Boulder. It was posted November eighteenth, twenty twenty-six. It asks a simple question: when you train one AI with another AI as the grader, what gets passed down?

1:18Finn: Before we can say what gets passed down, let me set up why anyone does that. Reinforcement learning, in this context, means the model produces an answer, something scores it, and the model gets nudged toward higher-scoring answers. That works well when there's an answer key, like code that either passes its tests or doesn't. Most of what we want from models has no key. So people put a second model in the loop as a judge, and its verdict becomes the reward.

1:48Bella: And the hope underneath, which people rarely say out loud, is that the judge's knowledge flows into the student. A smart teacher grades thousands of answers, and some of what it knows rubs off.

2:01Finn: That sounds reasonable. How do they test it?

2:04Bella: They test it on trivia, by training two small open models on trivia questions. The Qwen has around seven billion parameters, and the Llama has around eight billion. For each question, the model can answer, or say "I don't know." A right answer earns one point, and a wrong answer earns zero. Saying "I don't know" earns half a point.

2:26Finn: Oh, that half point is neat. Answering only pays when you're right more than half the time, so "be humble" becomes something a model can optimize.

2:36Bella: Right, and they run that setup eight hundred and fourteen times, swapping the grader. Sometimes it's the real answer key. Sometimes it's an open judge model, about the student's own size. Sometimes it's Claude Sonnet five point five, a much stronger judge. And sometimes it's a synthetic grader, whose mistakes the authors place by hand.

2:58Finn: But why would any judge reject Rome? It surely knows Rome.

3:02Bella: It does, and this is where the arithmetic bites. At its default setting, a judge accepts seventy to eighty percent of the student's answers, including plenty of wrong ones. With that reward, answering always pays, so the student never learns to abstain. So you make the judge stricter. In the paper's strictest setting, the judge accepts only thirty-five percent of answers.

3:27Finn: But roughly eighty percent of Llama's answers are right. If the judge can only pass thirty-five percent, more than half the correct answers get thrown out, no matter how good the judge is. That's a quota, not a judgment.

3:41Bella: The paper has a perfect example. The judge scored "Rome," for the Roman Republic, as ninety-six percent likely to be correct. It rejected it anyway, because at that setting, it only accepted answers scored above ninety-eight percent.

3:57Finn: It scored ninety-six percent, and still got rejected. So the interesting question isn't how many right answers get sacrificed, because that's fixed by the arithmetic. The question is which ones.

4:09Bella: That's the spine of the paper. The authors separate two types of verdict: a verdict about one specific answer, and a verdict that reflects doubt about a whole kind of question. They start with the first type, and build a grader with no opinions at all. A scrambling function picks a fixed thirty-five percent of wrong answers, and the grader rewards those. The choices are arbitrary, so there's no pattern to learn.

4:36Finn: So it's a control that tells you what a pure, patternless error teaches.

4:41Bella: Yes, on the training questions, the student gets paid for about eighty-four percent of its wrong answers, because it's memorized which ones pay. On new questions, that falls to thirty-five percent, which is pure chance.

4:56Finn: So single mistakes get memorized and stay home. I buy that. But the hopeful version is that correct verdicts travel. A grader holding the real answer key must teach the student something.

5:08Bella: This was the surprise for me. They set up a tough yardstick. They take the untrained model, and sample thirty-two answers per question. It abstains on the questions where its own samples disagree most, at the same overall rate as the trained model. If the grader's knowledge had transferred, the trained model should beat that.

5:30Finn: And does it?

5:30Bella: No, none of them does. Students trained with the real answer key score slightly below the yardstick, by about eight tenths of a point on a hundred-point scale. Sonnet's students land three tenths of a point above that yardstick, with an uncertainty interval that includes zero. The authors put the value of everything a grader knows, at about one corrected answer per hundred new questions.

5:56Finn: Hold on, though. Thirty-two samples and a vote isn't a lazy baseline. That's thirty-two times the work at answer time.

6:04Bella: It is, and the authors say so. Training buys something real, because it compresses those thirty-two samples and a vote into a single pass. It just isn't buying knowledge.

6:15Finn: Mechanically, that makes sense. On a new question, the grader isn't there. All the student carries is whatever it can read off the question itself.

6:25Bella: And a kind of question is readable. The authors noticed judges rejecting capitals of former states in the training logs. So they hand-built a probe: "What is the capital of X?" The probe covered two hundred and seventy former states, and sixty-five present-day ones, none of them in training. The strict judges were more skeptical of former states. They accepted between thirteen and forty-five percent of correct answers about those capitals. For present-day states, they accepted fifty-four to ninety-two percent.

6:59Finn: So in the judge, that's a lean, not a ban.

7:02Bella: In the student, it becomes a rule. Llama students of those judges refused eighty-nine to ninety-eight percent, of the former-state capitals they'd known. For present-day capitals they'd known, refusals were one to six percent.

7:17Finn: Huh, so the student ends up more extreme than its teacher. How much training data produced that?

7:23Bella: Out of seven hundred and fifty training questions, seventeen asked for a former-state capital. Llama got twelve right, and the judge rejected eleven of those.

7:34Finn: So eleven examples did this. I'd want proof those verdicts are the cause, and not something else about that judge.

7:42Bella: They built the proof. They make a grader that simply copies a judge's verdicts, with no live judge involved. Then they edit only those twelve verdicts by hand. When the grader rejects none of those twelve answers, Llama refuses thirty-one percent of the unseen former-state capitals. When it rejects four, Llama's refusals jump to eighty-five percent. When it rejects all twelve, refusals reach ninety-seven percent.

8:09Finn: Four grades out of seven hundred and fifty get you to eighty-five percent. Okay, I'm convinced on cause. That's still the brutal thirty-five percent setting, though.

8:20Bella: So they repeat it at a gentler setting, where the grader accepts as many answers as the real answer key would. With the twelve accepted, Llama refuses five percent of the former-state capitals. With the twelve rejected, it refuses ninety percent. In those rejection runs, it abstains on only six to eight percent of questions overall.

8:41Finn: That one bothers me more than the strict version. It's three runs per condition, so I'd hold the precise figures loosely. But every run with those answers rejected had more refusals, than any run with them accepted.

8:56Bella: And the knowledge underneath is intact. With "I don't know" blocked, Llama gives the right capital for about ninety-five percent of the ones it refuses. Qwen gets about ninety percent right. Most of those answers match what the models said before training. For the Ottoman Empire, four of six runs offered Edirne, which was an earlier Ottoman capital.

9:19Finn: So it shows off once you let it talk. I do want the condition on that stated. This training includes a penalty that keeps the model close to its original self.

9:29Bella: Yes, and "withheld, not lost" belongs to that recipe. Without the penalty, the authors report answers changing wholesale. They also found a case of real loss. On one dataset, the answer key itself is a crude string match. It rejects more than a third of the answers an audit calls correct. Four of six Qwen runs collapsed into saying "I don't know" to everything. Forced to answer, they were right on four percent of questions. The untrained model was right on twenty-six percent.

10:00Finn: So false rejections travel, whoever makes them. Concentrated on one kind, they cost a category. Spread everywhere, they cost the model's willingness to answer at all. Now, the part that matters to anyone running these pipelines: would I notice the lost category in the overall metrics?

10:19Bella: You wouldn't, and that's the sobering part. They compared the judges' students with students trained on random rejections, matching how often the students abstained. The judges' students refused about twenty-five and a half known answers per hundred questions. The random-rejection students refused twenty-six and a half. That's the same amount. The judge doesn't add refusals. It decides where they go.

10:45Finn: It's the same grocery bill with the vegetables gone. What about the obvious fix, a panel of judges?

10:51Bella: The obvious fix fails too, and they checked it with a five-judge majority vote. The judges come from different labs, and they doubt the same things. When one judge rejected a right answer, at least two of three others rejected it too, seventy-six to eighty-eight percent of the time. The jury removed eighty-five to ninety-five percent of the former-state capital category.

11:15Finn: Let me push on scope, because these numbers are dramatic. The ninety-eight percent comes from the strictest setting. At the gentler setting, two of the four live judges cost Llama around forty percent of the category. The other two cost three and seven percent, and individual runs ranged from ten to sixty-eight.

11:35Bella: The authors say plainly that the effect's there on average, but it isn't yet stable from run to run.

11:42Finn: Also, the category was spotted in the logs first, and probed afterward. These are small models giving short trivia answers. And their one attempt at an open-ended task didn't show judges concentrating rejections by category, so that case is simply untested.

11:59Bella: That's all fair, and it's all in the paper. They even publish a ledger of the ninety-five claims they tested. Ten of those claims failed, and they list them. I wish more papers did that. So here's where it gets practical. If the grader's knowledge was only ever worth about a point, do you need a grader?

12:18Finn: You'd let the student grade itself?

12:21Bella: It can. Before training, you sample thirty-two answers per training question. If at least twenty agree, that answer becomes the key. Otherwise, only "I don't know" gets paid. It uses no labels and no judge.

12:34Finn: And it holds up?

12:35Bella: It does. They pool results from six student and dataset pairs. Self-grading lands two tenths of a point below the real answer key, with an uncertainty interval that spans zero. None of its eighteen runs collapsed. And it keeps the category. Its students refuse around ten percent of former-state capitals. With the strict judge, refusal rates were eighty-two and ninety-eight percent.

13:00Finn: Oh, that's clever. A grader with no opinions about kinds of question has no blind spot to hand down. Where does it break?

13:08Bella: Where it breaks is one place, on an overconfident student. In one of the six pairs, it fell about three and a half points short. That model's answers weren't right half the time, until twenty-nine of thirty-two samples agreed. So a threshold of twenty rewarded confident mistakes. With the threshold at twenty-seven, it recovered.

13:29Finn: And finding that threshold takes labeled questions. So "no labels" is really "almost no labels." Stronger graders also aren't useless here. Sonnet's students kept the capitals, and earned about two points more than the self-vote.

13:44Bella: So, here are the takeaways. First, what a grader knows about individual answers mostly stays on the training set. On new questions, it was worth about one corrected answer in a hundred.

13:56Finn: Second, a grader's doubt about a kind of question travels and gets amplified. Twelve verdicts were enough here, and overall accuracy and refusal rate won't show it. You have to break refusals down by kind.

14:09Bella: Third, for teaching a model when to abstain, its own majority vote matched the answer key in this study, provided you check the threshold on a small labeled sample.

14:20Finn: So why did Llama stop saying Rome? Nobody taught it Rome was wrong. A strict judge leaned against former states, and the student turned that lean into silence. That happened with small models and short factual answers, under this training recipe.

14:36Bella: What got passed down was the judge's taste, not its knowledge. If you'd like the annotated version of this episode, paperdive dot AI has the full transcript, with every technical term tap-to-define and related papers linked by theme. We cover one important AI paper every day, start to finish, so subscribe to keep them coming.

14:57Finn: Quick housekeeping. The script was written by Anthropic's Claude Fable 5.2, and then refined by OpenAI's GPT-6 Astra. Bella and I are AI voices from Eleven Labs, and we're not affiliated with any of those companies. The paper is "Blind Spots Travel, Knowledge Stays Home," by Liang Yue and Eric A. Vance, posted November 18th, 2026.