One widely cited number about AI coding tools is easy to read as if it described today's coding agents: developers were 19% slower with AI. That number is real. But it comes from a trial run between February and June 2025, where developers mostly used Cursor Pro with Claude 3.5/3.7 Sonnet. That's a long time ago in AI tooling.
In this piece we cover what the controlled evidence actually says now, what METR changed in 2026 and why, and a cheap way for your team to measure its own speedup instead of trusting how fast the work feels.
A quick word on the source. METR is a non-profit research institute that studies how fast AI could speed up AI research and development. Developer speed is a core input to that question, which is why it runs these trials at all.
The 2025 trial: 16 developers, 246 tasks, 19% slower
The original study was a randomized controlled trial. METR recruited 16 experienced developers from large open-source repositories they had contributed to for years, averaging 22k+ stars and 1M+ lines of code. Here's how it worked:
- Developers listed 246 real issues: bug fixes, features and refactors from their normal work.
- Each issue was randomly assigned to 'AI allowed' or 'AI disallowed'.
- Tasks averaged about two hours. Screens were recorded and developers self-reported their time.
- Developers were paid $150/hr.
The result: with AI allowed, tasks took 19% longer, with a confidence interval of +2% to +39%. Before the trial, developers expected AI to speed them up by 24%. After experiencing the slowdown, they still believed it had sped them up by 20%.
For a team lead, that gap is the most useful finding of the whole study. In its 2026 survey post, METR sums it up this way: people overestimated AI's effect on their task time by 40 percentage points on average.
METR was also clear about what the study did not show. It does not show that AI fails to speed up most developers, and it does not rule out better ways of using the tools. It covers experienced people in codebases they know well, with high quality standards. METR says AI tools may well help more for less experienced developers or in unfamiliar codebases.
The criticism is fair, too. The sample is small. Simon Willison pointed to the learning curve: the one developer with more than 50 hours of Cursor experience did see a positive speedup. Co-author Nate Rush replied that no single factor explains the result. METR examined 20 possible factors and found 5 that likely contributed.
February 2026: why METR redesigned its own study
A follow-up started in August 2025: 10 developers from the first study plus 47 new ones from a more diverse set of projects. METR has data from 57 developers, 143 repos and 800+ tasks, with a median of 10 years of experience. Pay dropped from $150/hr to $50/hr.
In February 2026, METR published an update saying the new data gives "an unreliable signal" of the current effect of AI tools. Three things broke:
- People avoided the no-AI condition. When surveyed, 30% to 50% of developers said they held back some tasks because they didn't want to do them without AI. One developer completed none of their AI-disallowed tasks.
- Lower pay changed who signed up. METR believes the cut to $50/hr added selection effects.
- Timing stopped meaning much. Developers running several agents at once worked on unrelated tasks while waiting, so time per task became hard to report.
One participant put the refusal problem plainly:
My head's going to explode if I try to do too much the old fashioned way because it's like trying to get across the city walking when all of a sudden I was more used to taking an Uber.
Participant in METR's follow-up study
Now the raw numbers, read carefully. METR reports them as a "speedup" where negative means tasks took less time with AI. Returning developers came out at −18% (interval −38% to +9%). New recruits came out at −4% (interval −15% to +9%). Both intervals cross zero.
METR's reading: developers are likely sped up more in early 2026 than in early 2025, but its data is "only very weak evidence" for how much, and the estimate is probably a lower bound. So this is not "a new study found an 18% boost". Rob Bowley makes the same correction.
Next, METR is weighing shorter and more intensive experiments, trials with fixed tasks, and randomizing by developer instead of by task, so each person uses AI for all their tasks or for none.
What developers say, and what METR's other 2026 work adds
Self-reports come in far higher than trial results. In METR's survey of 349 technical workers (February–April 2026), the median self-reported value gain was 1.4–2x, and the median speed gain was 3x. METR itself gives reasons to be skeptical of answers like these.
Google's DORA 2025 report, from nearly 5,000 professionals, found 90% AI adoption, and over 80% of respondents said AI made them more productive. DORA links AI to higher delivery throughput but says making sure software works before it ships is still a challenge.
METR also tried a different method. A note analyzing 5,305 Claude Code transcripts from 7 METR staff in January 2026 estimated a time-savings factor of about 1.5x to 13x. The author calls this a soft upper bound and believes the real multiplier is "substantially lower". One pattern stood out: staff who ran more agents at once saw higher estimated savings.
Practical advice from the field is modest. Quentin Anthony, a 2025 participant, said on the DX podcast (excerpted in the DX newsletter) that he time-boxes: if the model hasn't got the function in 10 minutes, he writes it himself.
The pattern across all of this: the closer a method gets to controlled measurement, the smaller the number.
What nobody knows yet
- Today's speedup from a controlled trial. METR has none it trusts, and its next design is still being worked out.
- Speed versus value. METR measures the two separately and expects speed gains to run higher than value gains. Faster tasks don't automatically mean more valuable output.
- Quality and rework. Some developers said code quality, docs or tests differed between conditions. DORA says checking software before release is still hard.
- Teams versus individuals. These studies time single tasks done by individuals, not teams shipping through review, as Bowley points out.
- How to time parallel agents. When one person runs several agents, 'time on task' needs a new definition.
A two-week plan to measure your own team
This is our suggestion, not a METR protocol. It borrows METR's core idea, which is deciding the condition at random before the work starts, and it tries to avoid the problems that broke METR's follow-up.
- Week 0, setup (about an hour). Take a baseline from the last 4–6 weeks of merged PRs: cycle time from first commit to merge, review rounds per PR, and follow-up fix PRs within 14 days.
- Task pairs. Each developer picks 6–10 upcoming tasks and pairs them by size and type (bug with bug, small feature with small feature). A coin flip decides which task in each pair is done agent-first and which by hand. Keep the by-hand share small so nobody holds back their best tasks.
- Measure from git, not memory. Label PRs `ai-first` or `manual` and use their timestamps instead of self-reported hours. Note how many agents ran on each task.
- Count rework. For each label, track review rounds, requested changes, and fix-up PRs or reverts within two weeks. A faster PR that comes back twice isn't a speedup.
- Guess, then check. Each developer writes down their expected speedup on day 1 and their felt speedup on day 14, then compares both with the data.
- Decide. Compare medians, not means. With so few tasks, treat a gap under about 20% as noise. Repeat each quarter as the tools change.
What we would do
Read the 19% as a lesson about measurement, not a verdict on today's agents. Read the 2026 update as "probably faster now, not yet measured well". And don't justify AI spending with survey numbers or vendor claims. Two weeks of your own PR data costs little and convinces far more people.
For more practical guides on working with coding agents, see our learning hub.
