On March 31 artificial intelligence company Anthropic accidentally leaked roughly 512,000 lines of code, and within hours, developers were poring over it. Among the surprises was code inside Claude Code, Anthropic’s AI coding assistant, that appears to scan user prompts for signs of frustration. It flags profanity, insults and phrases such as “so frustrating” and “this sucks,” and it appears to log that the user expressed negativity.
Developers also discovered code designed to scrub references to Anthropic-specific names—even the phrase “Claude Code”—when the tool is used to create code in public software repositories, making the latter code appear as though it was entirely written by a human. Alex Kim, an independent developer, posted a technical analysis of the leaked code in which he called it “a one-way door”—a feature that can be forced on but not off. “Hiding internal codenames is reasonable,” he wrote. “Having the AI actively pretend to be human is a different thing.” Anthropic did not respond to a request for comment from Scientific American.
The findings expose a problem emerging across the AI industry: tools that are designed to be useful and intimate are also quietly measuring the people who use them—and obscuring their own hand in the work they help produce. Anthropic, which has staked its reputation on AI safety, offers an early case study in how behavioral data collection can outpace governance.
On supporting science journalism
If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.
Technically, the frustration detector is simple. It uses regex, a decades-old pattern-matching technique—not artificial intelligence. “An LLM company using regexes for sentiment analysis is peak irony,” Kim wrote. But the choice, he notes in an interview with Scientific American, was pragmatic: “Regex is computationally free, while using an LLM to detect this would be costly at the scale of Claude Code’s global usage.” The signal, he adds, “doesn’t change the model’s behavior or responses. It’s just a product health metric: Are users getting frustrated, and is the rate going up or down across releases?”
Miranda Bogen, director of the AI Governance Lab at the Center for Democracy & Technology, says the more pressing issue is what happens to such information once a company has it. “Even if it’s a very legible and very simple prediction pattern, how you use that information is a separate governance question,” she says. A signal collected for one purpose can migrate into other parts of a product in ways users neither expect nor consent to.
Bogen says the pattern is familiar from older Internet platforms, where small behavioral cues became signals that shaped what users saw and how they were categorized. AI companies are reprising a similar privacy problem: users hand these systems enormous amounts of information precisely because the tools are designed to know them well enough to be useful. “Who is keeping track of things about users?” Bogen asks. “And how is that information being used to make determinations about them?” What the Anthropic leak made plain is that, at least at one company, such accounting is already written into the code.