Developers using AI tools took 19% longer to finish coding tasks, a METR study found, challenging assumptions about productivity gains in open-source work.
Artificial intelligence tools can slow down experienced open-source developers rather than accelerate their work, according to findings released by the nonprofit AI research group METR.
Developers and experts widely anticipated that using advanced tools would significantly improve productivity and shorten the time required to complete tasks in familiar codebases by approximately 24%; however, a randomized study by METR found that the opposite was true.
“Surprisingly, we find that when developers use AI tools, they take 19% longer” than without the cutting-edge tools, according to the study’s authors.Benchmarks ‘sacrifice realism’
The study also pointed out that “While coding/agentic benchmarks have proven useful for understanding AI capabilities, they typically sacrifice realism for scale and efficiency.”
The report’s authors go on to explain that coding tasks are often “self-contained” and do not require prior context to understand the tools’ capabilities.
