According to a study, AI coding assistance do not increase productivity or reduce stress.

Developers were anticipated to be significant beneficiaries of the generative AI trend, as specialized tools promised to enhance the speed and ease of code production. However, a recent analysis conducted by Uplevel, a company focused on coding metrics, indicates that these anticipated productivity improvements have yet to materialize.

The research monitored approximately 800 developers, assessing their performance with and without the assistance of GitHub’s Copilot over three-month intervals. Contrary to expectations, Uplevel’s evaluation of critical metrics such as pull request cycle time and throughput revealed no substantial enhancements for those utilizing Copilot.

Matt Hoffman, a data analyst at Uplevel, shared with the CIO that the initial hypothesis was that developers would produce more code and experience a reduction in defect rates due to the support of AI tools in code review processes. However, the results contradicted these assumptions, showing that developers employing Copilot actually introduced 41% more bugs into their code. Furthermore, Uplevel found no indications that the AI assistant contributed to alleviating developer burnout, challenging claims from GitHub and other advocates of AI coding tools regarding significant productivity increases.

Developers may be experiencing favorable outcomes, as early reports from Copilot indicated that nearly 30% of newly written code incorporated AI assistance, a figure that is likely to have increased. Nonetheless, there is a concern that this rise in usage may stem from programmers developing a reliance on these tools, potentially leading to a decline in their initiative and creativity.

The practical experiences with AI coding assistants have varied significantly across the industry. For instance, Ivan Gekht, CEO of custom software company Gehtsoft USA, expressed to the CIO that the AI-generated code often presents challenges in terms of comprehension and debugging. As a result, the team sometimes finds it more efficient to start from scratch rather than attempting to work with the AI-generated output.

Supporting Gekht’s perspective, a study conducted last year revealed that ChatGPT answered over half of the programming questions incorrectly, although it has since undergone substantial improvements through various updates. Gekht emphasized that software development primarily involves cognitive processes such as understanding requirements and system design, with the actual coding being the less complex aspect. In contrast, Travis Rehl, CTO of Innovative Solutions, reported remarkable productivity gains, with developers achieving up to three times their usual output through the use of tools like Claude Dev and Copilot. These contrasting experiences suggest that the field of AI coding assistants is still in its nascent stages, and their future development remains uncertain.

Thank you for reading this post, don't forget to follow my whatsapp channel