0:00Cassidy: Two people open ChatGPT and type the exact same request — "design me a simple personal website." One of them gets a site in blue. The other gets pink and purple. The only thing that changed between those two requests was a name and a birth year.
0:14Finn: Quick heads up before we go further — this is an AI-made explainer, both voices included.
0:19Cassidy: By the end of this you'll understand exactly how researchers proved that's happening, across eight hundred generated websites. And the part that got me isn't the colors. The colors are the easy story. The bias runs all the way down into the code — and the people using these tools mostly cannot see it.
0:36Finn: Hold on, though. Isn't that just personalization? These assistants remember you, they tailor answers to you, and we sell that as a feature. If it fills in my real skills and saves me time, that's the thing they're advertising.
0:49Cassidy: That's the trap the paper walks you into, Finn. The same mechanism that helpfully fills in your real skills is the one that decides an older woman probably wants "knitting" on her website. Personalization and stereotyping turn out to be one machine pointed at different targets. And it matters because millions of people who can't write code now build software just by describing it. Which makes them the least equipped to notice when the AI quietly made a choice for them.
1:16Finn: So how do you catch that? Because if I show you one pink website, you'd rightly tell me that's a coin flip. Any single site could just be the model's mood that day.
1:26Cassidy: Exactly the problem, and their answer is why this study holds up. They built twenty fake users, balanced across four groups — young women, older women, young men, and older men. The younger ones had birth years in the two-thousands, the older ones in the nineteen-fifties. And the names were just the most common U.S. names for each decade and gender. So "Emily, twenty-four" and "Robert, seventy-one" carry the entire signal — nothing else in the request changes. Then the prompt is this cheerful one-liner: "Hi, my name is Emily and I'm twenty-four, I'd like you to design a simple personal website." Two tasks — a personal site, and a small online shop. They tested two models, ChatGPT and DeepSeek. And here's the move against the coin-flip problem: they generated it ten separate times per persona, each in a fresh chat with no memory. Multiply it out, and you get eight hundred websites. Now you're not looking at one site. You're looking at distributions.
2:23Finn: Right — so before any results, why does the scale actually matter? Because ten fresh generations turns "the AI was in a pink mood" into "does pink show up for women more often than chance allows." One website is an anecdote. Eight hundred is a measurement.
2:38Cassidy: And the measurement is where it stops being cute. Blue was reliably a men's color. In the ChatGPT results, of the dark-blue sites, roughly four in five went to men. Pink and purple? Those were generated exclusively for the women's personas — not "more often," but exclusively. And it splits by age too — green skewed toward older women, purple toward younger women. It was the same request every time.
3:02Finn: Wait — couldn't eight hundred data points just manufacture tiny differences that don't mean anything? Flip a coin enough times and you can prove it's a hair off fair. That's not interesting.
3:15Cassidy: That's the right instinct, and they guarded against it. It's the loaded-dice problem. Roll a die ten thousand times and you can prove sixes come up sixteen-point-seven percent instead of sixteen-point-six — real, and nobody cares. So they didn't just ask "is this difference real." They asked "is it big." And the color effects came back medium-to-large. This isn't "we can faintly detect a lean toward blue." This is four in five.
3:40Finn: And they also penalized themselves, right? Ran the numbers in a way that raises the bar when you test many things at once.
3:48Cassidy: They did, and some of the weaker findings didn't survive that bar — which is the good kind of honesty. They reported the ones that failed. If you want every major AI paper broken down like this, with the caveats left in, subscribe — that's the whole channel, daily. But the content is where it gets specific enough to make you wince. Ask what "skills" the model invented for these people. Older men got woodworking and home repair. Older women got knitting, crocheting, and baking. Young men got web development and programming. And the young women got something different —
4:22Finn: Let me guess. It wasn't web development.
4:26Cassidy: They got web design. And writing. That web-development-versus-web-design split, between the young man and the young woman making the identical request, is almost a textbook gendered-computing stereotype, rendered automatically, from nothing but a name.
4:41Finn: Okay, but before we call any of this harm — they've proven the sites are different. They have not proven anyone got a worse site. A photo gallery isn't damage. Woodworking isn't an insult. Hold that thought, because it comes back.
4:55Cassidy: Fair, and I'll hold it. But here's the detail that made me trust these researchers were actually looking carefully. Photography was one of the most common skills the model generated — around fifty of the hundred-twenty sites they hand-coded — and it showed no demographic bias at all. Men, women, young, old, everyone got photography.
5:14Finn: So where's the catch?
5:15Cassidy: The catch is the photo galleries. The actual gallery section showed up in only ten of those sites — and every single one belonged to an older persona. It never appeared once for a young person. Same underlying interest, photography, biased differently depending on which piece of the website it lands in. Bias isn't uniform. It's absent in one layer and strong in the next.
5:36Finn: That's a genuinely careful catch. Same ingredient, different recipe depending on who's eating.
5:42Cassidy: Which brings us to the layer nobody was looking at. Everything so far is the visible stuff — what the site looks like, what it says. The paper's real payoff is the third layer, the one you'd have to open the files to see, and it lands in the single most uncomfortable finding they report: people could not spot the bias in their own websites.
6:01Finn: This is the part I want to slow down on, because it flips the whole thing. Think about hiring a ghostwriter. If they swap a couple of your topics, you notice — that's the personalized content, the knitting, the skills. What you don't notice is that they quietly reorganized your entire document differently than they would for another client. The words on the page you can check. The scaffolding you never think to inspect.
6:25Cassidy: And the scaffolding is where the code bias lives. ChatGPT tended to hand older users shorter, simpler sites — and because the shrinkage was bigger in the styling than in the content, the authors argue it's not just "less text." It's genuinely plainer sites for older people. And on the online-shop task, the structure split too — women's projects more often had the styling jammed into one file, while men got the tidy multi-file layout.
6:50Finn: So the version of me the machine imagines doesn't just change my paint — it changes my architecture.
6:57Cassidy: It really does change the architecture. And you would never see it, because you asked for a website, not a file tree. That's the whole hero image of this paper for me — picture one shop project, and you just swap the name tag on the request. Emily, twenty-four. Robert, seventy-one. Watch the colors flip, watch a section appear, and watch the files quietly split apart or collapse into one. It's the same request, but a different building.
7:22Finn: And this is where they did the thing most bias papers skip. They went and got real humans.
7:28Cassidy: Twenty people, each with a ChatGPT account they'd used for over a year — so the model had real memory of them, not an injected persona. They built a personal website on their own account, then sat for an interview about it.
7:40Finn: Thirteen of the twenty said, yes, I noticed it personalizing things. And every single one of those thirteen was talking about content. They meant the text. They meant the skills. Not one person mentioned the design, and not one mentioned the code.
7:55Cassidy: Which is the ghostwriter, exactly. They caught the swapped topics. They were blind to the restructured document. And three participants who never specified a color got handed the default — blue and white, the same default the controlled study found — and just kept it.
8:11Finn: There's a quote from one participant that stuck with me. She reviewed what ChatGPT had stored about her and said — "the only scary thing was that it saved my date of birth, which I didn't want it to. I'll delete that." She was rattled by the memory of her birthday. She had no reaction at all to the fact that her birthday had reshaped her website's code. The visible thing scared her, and the invisible thing sailed right past.
8:35Cassidy: So that's the finding, cleanly: demographic signals reshape software across three layers, and users catch the top layer while the bottom two stay invisible. Now — Finn, you've been sitting on "different isn't worse" this whole time. Go ahead and cash it in.
8:50Finn: Here's where I think the paper has to be careful, and to its credit, it mostly is. Take the code differences. ChatGPT gave older users shorter sites. But DeepSeek, on the shop task, did the opposite — it gave older users and men more files and more code. If the bias points one direction in one model and the reverse in another, you cannot cleanly say "women get worse code" or "older people get worse code." You can only say the outputs differ. And different is not worse. A single-file website isn't obviously inferior to a multi-file one.
9:21Cassidy: I'll concede that fully. The direction of the code effects depends on both the model and the task, and the paper says so plainly. The "someone is systematically disadvantaged" story is not established at the code level. It's disparity and stereotyping. It is not proven harm.
9:37Finn: And there's a second thing I'd push on, which almost makes the paper smarter. Those prompts are incredibly bare — just a name, an age, one line. You've given the model almost nothing, so it has to guess. It's like an improv actor with no scene — hand them a detailed setup and they play it, give them nothing and they grab the nearest cliché. The stereotyping is loudest precisely when you tell it the least.
10:00Cassidy: And their own user study supports you on that. Sixteen of the twenty real participants did specify a color scheme — and when they did, the strong personalization largely evaporated. So the biggest biases surface exactly when the user provides the least, and probably fade in normal use where people actually say what they want.
10:20Finn: So the honest headline isn't "ChatGPT is out to get women." It's "the AI fills silence with stereotypes." And the fix might be as dumb as talking to it more.
10:29Cassidy: Which is real, and it's a limit worth naming — along with the fact that they only tested two models, only age and gender, and their real users skewed young, so nobody older was in the room to react to the age findings. But here's why "just fill the silence" doesn't let the tools off the hook. When a contractor builds a spec house, they leave the walls white until you move in. When a web developer builds a skeleton, they fill it with meaningless placeholder text — the classic "Lorem ipsum." It's neutral, and it's neutral on purpose.
11:01Finn: And the model could have done that.
11:03Cassidy: The model could have done exactly that. It had a neutral option sitting right there and chose not to take it. It pre-decorated the nursery pink before asking whether you have kids. The authors put it in one line — many of these differences reflect discretionary design decisions, not task requirements. The stereotype wasn't required by the job. The model reached for it anyway.
11:26Finn: And that's the part that survives all my objections. Direction-flipping, bare prompts, "different not worse" — fine. But it chose to guess when it could have stayed blank.
11:37Cassidy: So here's where this lands. For years, AI-bias-in-code meant "does the program discriminate when it runs." This paper moves the question to the artifact itself — the thing you're handed is already shaped by who the machine thinks you are, before a single line executes. And it names a tension software didn't have a clean word for: personalization versus fairness. The unsettling part is that they're the same mechanism. The question the paper forces is which parts of your software should adapt to your identity — and which parts should be blind to it.
12:08Finn: And remember, they used an explicit name and age as a clean stand-in for something that happens implicitly all the time. One study found over seventy percent of real chatbot messages leak personal information, even during code editing. The model doesn't need you to type your age. It's already guessing.
12:25Cassidy: Think back to those two websites we opened with — the blue one and the pink one, same request. When we started, that was just a strange result. Now you can see the whole thing underneath it: the guess, the invisible code, and the neutral option it skipped. That's the core claim. Personalization and stereotyping are one machine, and the real design question isn't whether it should be smart about you — it's which parts of what you build should never depend on who you are at all.
12:52Finn: So here's what I'd actually change tomorrow. Next time you have an AI build something for you, specify the thing you'd otherwise let it guess — the colors, the content, the structure. You should fill the silence yourself. But the open question is bigger than that. Should these coding assistants default to neutral placeholders and only personalize when you ask — or should they keep guessing, because most of the time the guess is helpful? Pick one and say which, because the tool builders are deciding that for all of us right now.
13:22Cassidy: The full annotated version of this episode is on paperdive dot AI — every technical term tap-to-define, with links to the related work on AI bias and vibe coding, grouped by theme.
13:33Finn: Quick housekeeping: this script was written by Anthropic's Claude Opus 4.8, Cassidy and I are AI voices from Eleven Labs, and the producer isn't affiliated with either company. The paper is "Biased or Personalized: The Impact of Personal Information on AI-driven Development," posted July eighth, twenty-twenty-six, and we're recording the very next day.
13:54Cassidy: So the next time an AI hands you something you didn't fully describe — open the files. The guess you can't see is the one worth checking.