Harvard researchers interviewed engineers and managers as part of a study into how software work changed after companies adopted AI. One junior software engineer said people could produce a great many commits quickly, but code review remained a bottleneck. On one project, only one senior engineer knew the codebase well enough to approve changes. That person was so busy they could review the junior engineer’s code only once a week. The numbers the researchers found in work records from 718 companies point in the same direction. After companies adopted coding agents, code output rose 30%, but there was no significant increase in the work they completed.
The comparison was with autocomplete AI, not with no AI
Harvard researchers Fiona Chen and James Stratton analyzed records from 725,938 people at 718 customers of development analytics company Jellyfish that had consented to research use. The records span January 2021 to March 2026 and include about 68 million Jira issues, 200 million commits, and 49 million PRs, or requests to merge code changes. The researchers did not examine the code inside commits. They looked only at records of who submitted and merged what, and when.
The comparison took advantage of differences in when companies adopted the tools. At one company, a single purchase involved four teams: sourcing, legal, security, and IT. It did not buy a few trial licenses until fall 2023, almost a year after ChatGPT launched in November 2022. Another company said approval from IT and legal usually took less than a day. The researchers used a staggered difference-in-differences approach, comparing early adopters with companies that adopted later or not at all, and controlled for company size.
The baseline matters. The researchers estimated the effects of code autocomplete AI assistants such as Copilot and Cursor separately from those of agents such as Cursor Agent, Claude Code, and Devin that carry on tasks autonomously. The agent effect is the additional change from adding agents to a workplace already using assistants, not the change from having no AI at all. The share of companies using agents rose from nearly 0% in October 2024 to more than 95% in January 2026.
Adopting assistants alone was associated with a roughly 5% increase in PRs, which was not statistically significant. The study is a job market paper submitted by its authors as part of the academic hiring process, and there is no indication that it has been peer reviewed. The data came from Jellyfish, which sells development productivity metrics. The paper says its conclusions are the authors’ own, and that Jellyfish checked only for disclosure of proprietary information and did not review the scientific content.
“Completed work” meant work reviewed, tested, and deployed to production
The researchers measured work completed by a company through the number of Jira issues resolved. In this paper, a resolved issue means the code has been reviewed, tested, and deployed to production. No matter how much code is written, it does not count unless it passes review and is deployed. Engineers resolved an average of 3.67 issues per month, and the estimated effect of agent adoption was +0.12 issues, with a standard error of 0.17. That estimate is not distinguishable from zero. There was no significant effect on epics, which represent features or projects, either.
The paper says its 95% confidence interval rules out effects greater than 12% of the baseline average. At the same companies where code output rose 30%, the increase in work completed through deployment, if any, was below 12%.
It is possible that each issue became larger while the count stayed flat. The researchers checked this in two ways. First, they used GPT-4o to estimate the time needed for 0.5% of issues, then trained a model to extend those estimates to all issues. Second, they used the expected time needed to complete an issue, known as predicted cycle time. They found no evidence that the mix of issues had changed. The researchers also questioned lines of code and commits as metrics in their own right.
“Many managers rely on intermediate productivity metrics, such as lines of code or commits; however, these may not accurately capture the true gains from AI adoption.”
In other words, many managers look at intermediate metrics such as lines of code or commits, but those figures may not accurately capture what AI adoption actually achieves.
The time from submitting a code change to merging it after human review rose from 7 days to 10.5
To see where the additional code accumulated, the researchers measured the days from PR submission to merge. After agent adoption, that period rose by 3.45 days from an average of 7.03 days to about 10.5 days, with a standard error of 1.60. The increase is statistically significant, but the estimate has a wide range of uncertainty. At companies using assistants alone, there was no significant change in this period.
More precise estimates came from the contents of reviews. The share of PRs receiving change requests and the number of review comments per PR rose substantially, while PR size did not change significantly. So the results do not support the explanation that comments increased because PRs became larger.
| Metric | Baseline average | Effect of agent adoption | Interpretation |
|---|---|---|---|
| PR submission to merge | 7.03 days | +3.45 days (standard error 1.60) | Significant, wide uncertainty |
| Share of PRs receiving change requests | 0.13 | +0.12 (standard error 0.01) | Nearly doubled |
| Review comments per PR | 1.66 | +0.58 (standard error 0.11) | Up 35% |
| Lines added per PR | 1,167.97 | −67.99 (standard error 126.91) | Not significant |
The junior engineer quoted earlier added in the interview:
“After some outages that were caused by AI-generated commits, there were also new rules requiring senior engineer approval for AI-generated code, which created even more review bottlenecks.”
After several outages caused by AI-generated commits, the company introduced a rule requiring senior approval for AI-generated code, making the wait for review even longer. The authors said they lightly edited the quotation for clarity.
More change requests do not prove that bugs increased. The paper’s model interprets the change as possibly arising from lower draft quality, the “bug-rate channel,” but the paper provides no counts of bugs, reverts, or outages, only interview anecdotes. That interpretation also differs from the data provider’s promotion. In a January 2026 blog post about a preliminary version of the study covering 500 companies and 100,000 people, Jellyfish wrote that “code quality doesn't seem to be suffering”. A counterargument on Hacker News held that quality was the bottleneck all along, with review merely where the quality problem became visible.
Within the same workforce, the share of engineers doing reviews rose from 0.29 to 0.33
Responsibilities shifted within companies. The share of engineers performing reviews rose by 0.04 from 0.29, an increase of 14%, while the share writing code, 0.43, did not change significantly. Employment levels did not change significantly either. The estimates ruled out a decline greater than 2.9% in total employment and greater than 13.8% in engineering employment after agent adoption, and showed no significant change for either junior or senior engineers. The result points to a larger share of the existing workforce taking on reviews, without additional hiring.
AI code review tools also spread quickly. By March 2026, nearly 80% of companies in the sample had tried them. Yet AI wrote 23.3% of all review comments, and 10.8% of PRs received at least one AI comment. Humans wrote the remaining 76.7% of comments. The paper concludes that the review bottleneck persists even after AI review tools are introduced.
One developer on Hacker News wrote that senior engineers find it very hard to give up line-by-line review. The developer argued that things would play out differently at startups that never introduce human review in the first place, and that such organizations are unlikely to be well represented in a sample like this one. The paper’s public description does not separately analyze this point.
AI sped up coding, but human review now sets the pace of completed work
A senior engineer quoted in the paper’s appendix described the arrangement this way:
“There is only one bottleneck at a time. If you speed up coding, the bottleneck shifts somewhere else.... You can generate features much faster now, but they still need to be validated and approved by humans.”
Estimates from Jellyfish records show engineers spending 42.0% of their working time writing code, 18.6% reviewing, and 39.5% on administrative work. Agents sped up the coding portion. But code must pass human review before it can be deployed. As code output grew, so did the work involved in each review. PRs receiving change requests nearly doubled, and review comments per PR rose 35%. Within companies, the share of existing staff doing reviews grew 14%: those people shifted into review rather than being newly hired.
It is still unclear whether moving people into review is enough. The paper argues that if the bottleneck comes from draft quality, shifting people to review alone will not solve it and the technology itself must improve. Without bug records, this paper alone cannot determine whether the near doubling of PRs receiving change requests signals a quality problem. Either way, the additional code piled up at human review.
The findings have a time limit. The data ends in March 2026, and Ars Technica noted that agents have improved substantially since then and that many companies are still learning how to use them. The proprietary data cannot be independently reproduced, and agents used through personal licenses are not captured in the adoption records.
Many managers have focused on how much code engineers write. Coding agents increased lines of code by 30%, but work completed through deployment did not increase significantly. Managers should now focus on how quickly that code passes human review, and how many review rounds it takes.
For a closer look
Artificial Intelligence in the Firm: Bottlenecks in Software Production The original paper by Chen and Stratton. Section 4 analyzes the review bottleneck, and Appendix B contains engineer interviews. fion.ac
Harvard researchers find AI coding agents shifted the bottleneck to code review A critical reading of the paper’s relationship with Jellyfish and the difference between promotion of the preliminary version and interpretation of the current one. complexdiscovery.com
AI coding agents generate more code, but not more software An article presenting figures and numbers from the paper and discussing the limits imposed by the age of the data. arstechnica.com