AI tools are revolutionizing coding and IT work, but are they making developers faster? One researcher says no. Find out why AI sometimes slows experts down and speeds up mistakes.
Our story begins, as many stories do, with a man and his AI. The man, like many men, is a bit of a geek and a bit of a programmer. He also needs a haircut.
The AI is the culmination of thousands of years of human advancement, all put to the service of making the man’s life a little easier. The man, of course, is me. I’m that guy.
Unfortunately, while AI can be incredibly brilliant, it also has a propensity to lie, mislead, and make shockingly stupid mistakes. It is the stupid part that we will be discussing in this article.
Anecdotal evidence does have value. My reports on how I’ve solved some problems quickly with AI are real. The programs I used AI to write with are still in use. I have used AI to help speed up aspects of my programming flow, especially when I focus on the sweet spots where I’m less productive and the AI is quite knowledgeable, like writing functions that call publicly published APIs.
You know how we got here. Generative AI burst onto the scene at the cusp of 2023 and has been blasting its way into knowledge work ever since.
One area, as the narrative goes, where AI truly shines is its ability to write code and help manage IT systems. Those claims are not untrue. I have shown, several times, how AI has solved coding and systems engineering problems I have personally experienced. AI coding in the real world: What science reveals
New tools always come with big promises. But do they deliver in real-world settings?
Most of my reporting on programming effectiveness has been based on personal anecdotal evidence: my own programming experiences using AI. But I’m one guy. I have limited time to devote to programming and, like every programmer, I have certain areas where I spend most of my coding time.
Recently, though, a nonprofit research organization called METR (Model Evaluation & Threat Research) did a more thorough analysis of AI coding productivity.
Their methodology seems sound. They worked with 16 experienced open-source developers who have actively contributed to large, popular repositories. The METR analysts provided those developers with 246 issues from the repositories that needed fixing. The coders were given about half the issues where they had to work on their own, and about half where they could use an AI for help.
The results were striking and unexpected. While the developers themselves estimated that AI assistance increased their productivity by an average of 24%, METR’s analytics showed instead that AI assistance slowed them down by an average of 19%.
That’s a bit of a head scratcher. METR put together a list of factors that might explain the slowdown, including over-optimism about AI usefulness, high-developer familiarity with their repositories (and less AI knowledge), the complexity of large repositories, lack of AI reliability, and an ongoing problem where the AI refuses to use „important tacit knowledge or context.“
I would suggest that two other factors might have limited effectiveness:
Choice of problem: The developers were told which issues they had to use AI help on and which issues they couldn’t.
