METR had to redesign their developer productivity research. 30–50% of invited developers refused to participate. The reason: they wouldn't code without AI.
This might sound like evidence of productivity gains. But the productivity numbers METR actually measured pointed in the exact opposite direction.
Everyone believes this — "AI makes me 2x more valuable"
Developer perception of AI is strong. METR study participants reported on average that AI made them twice as valuable to their organization, with a self-reported ~20% productivity boost. Problems getting unblocked faster, code shipping quicker — that's a real feeling.
This isn't entirely wrong. Finishing a function in 5 minutes that would've taken 30 without AI is genuinely faster. The measurement trap lives elsewhere — the maintenance cost of that 5-minute AI-generated code over the next few weeks doesn't register in the moment.
From a METR study participant
"My head's going to explode if I try to do too much the old fashioned way because it's like trying to get across the city walking when all of a sudden I was more used to taking an Uber."
But the numbers say the opposite
In METR's early 2025 randomized controlled trial, experienced developers were split into AI-assisted and unassisted groups, and task completion time was measured. The result: AI-assisted developers took an average of 19% longer to complete tasks than those without AI. Self-reported +20% vs. measured -19%. That gap is the core of the AI productivity illusion.
Code quality tells an even starker story. CodeRabbit analyzed 470 open-source GitHub pull requests and found that AI-assisted PRs contained 1.7x more issues than human-written PRs. That's 10.83 issues per PR vs. 6.45. Security vulnerabilities (XSS) appeared 2.74x more often, logic errors 75% more, performance regressions a staggering 8x more.
| Issue Type | AI-Generated Code | Human-Written Code |
|---|---|---|
| Avg issues per PR | 10.83 | 6.45 |
| XSS security vulnerabilities | +174% more | baseline |
| Logic & correctness issues | +75% more | baseline |
| Readability issues | +200% more | baseline |
| Performance regressions | +700% more | baseline |
The cost shows up fast. According to Entelligence, an AI coding analytics firm, 44% of the AI tokens companies spend go toward fixing bugs that AI generated in the first place. You use AI to boost productivity, it generates bugs, and you use more AI to fix those bugs.
At company scale it's even clearer. Uber implemented an AI usage leaderboard to encourage adoption — and burned through its entire 2026 AI budget within four months. COO Andrew Macdonald's conclusion: "It's very hard to draw a line between one of those stats and actually producing 25% more useful consumer features."
Amazon made a similar mistake. They launched Kirorank, an internal AI usage leaderboard, only to discover employees were running pointless fake tasks through AI to climb the rankings — dubbed "tokenmaxxing." Amazon shut it down in May 2026. Their message to employees: "Don't use AI just to use AI."
The scarier part comes next — silent skill atrophy
There's a longer-term problem beyond the short-term numbers. Refusing to work without AI isn't just a preference — it may signal that the ability to evaluate code without AI is actively diminishing.
Programming skill is "procedural memory" — the kind built only through repeated cycles of writing, failing, and fixing code yourself. When AI skips that loop for you, your ability to judge what code is good or bad weakens. This is neuroscientifically grounded: actively generating something activates multiple brain regions simultaneously, forming durable long-term memory.
The critical paradox: the developers who use AI best are those who can evaluate code without it. Reviewing AI-generated code, designing architecture, catching security flaws — these require judgment AI cannot provide. Right now, many developers are handing off the very experiences that build that judgment.
"Developers are trading temporary speed gains for permanent indenture — unless producing code faster halves the maintenance burden."
— James Shore, software engineering author
So how should you actually use it?
The answer isn't to stop using AI coding tools. Singapore Management University researchers and industry practitioners converge on this — treat AI like a senior engineer treats a junior: review, understand, and own the output.
- Make AI code review non-negotiable
AI-generated PRs must be reviewed line-by-line before merging. "AI wrote it" is never a reason to skip review. - Schedule deliberate AI-free coding time
Once a week, solve a small feature or bug without AI. The more uncomfortable it feels, the more you need it. - Measure deployment quality, not token usage
Don't repeat Amazon or Uber's mistake. Bug rate, post-deploy incidents, and PR cycle time are real signals. - Treat AI code like junior developer code
Before merging anything AI wrote, ask: "Can I explain why this code works?" If not, don't ship it. - Keep architecture and security human-owned
There are things AI genuinely struggles with — codebase-wide architectural decisions, org-specific security policy, legacy integration. Design these yourself.
Even the creator of Devin said it
Scott Wu, CEO of Cognition (maker of AI coding agent Devin), acknowledged that current AI coding agents operate at "junior to mid-level programmer" skill levels. That's why you can't delegate senior-level judgment to AI.
Want to go deeper?
METR Developer Productivity Study Update (Feb 2026) Why the research team had to redesign their experiment — the developer refusal cases and current limits of AI productivity research. metr.org
State of AI vs Human Code Generation Report CodeRabbit's primary data comparing bug density and security vulnerabilities across 470 open-source PRs. coderabbit.ai
Uber COO: AI spending is harder to justify Andrew Macdonald's comments and the full story of the four-month budget burn. techspot.com
Why Amazon shut down Kirorank The full story of the tokenmaxxing incident and what metric Amazon pivoted to after. mlq.ai
Developers Who Stopped Growing in the AI Era An in-depth analysis of how AI dependency affects developer growth through the lens of procedural memory and neuroscience. evan-moon.github.io
2025 was speed. 2026 will be quality. CodeRabbit's industry forecast on AI code quality as the next competitive differentiator. coderabbit.ai
