Two kinds of evidence get mixed together in vendor decks. A controlled lab task can show large speedups. Field experiments on real engineering work show smaller, messier gains. Both can be true. They answer different questions.
The lab result
Peng, Kalliamvakou, Cihon, and Demirer (arXiv 2302.06590) ran a controlled experiment: developers implemented an HTTP server in JavaScript as quickly as possible. The Copilot group finished about 55.8% faster than the control group (95% CI roughly 21–89%).
The task is standardized and short. That is why the estimate is precise—and why it should not be read as “shipping software gets 56% cheaper.” Collaboration, review, and product ambiguity are mostly absent.
The field experiments
A separate Microsoft–Accenture field project randomly offered GitHub Copilot to developers and tracked pull requests per week. Preliminary results on mit-genai.pubpub.org report roughly 12.9%–21.8% more PRs per week at Microsoft and 7.5%–8.7% at Accenture, depending on specification.
The authors are explicit about limits: low compliance at Microsoft and organizational changes at Accenture made estimates imprecise. Significance often required focusing on periods where uptake differed most between treatment and control.
Why the gap is expected
Lab tasks remove meetings, unclear requirements, review cycles, and flaky environments. Field PR counts include all of that. A tool that drafts code faster can still leave the bottleneck in review, testing, or product decisions.
PR volume is also an imperfect productivity measure. More PRs can mean faster delivery—or smaller, noisier diffs. Pair any rollout with acceptance criteria, change-failure signals, and review load.
How to use both numbers
Use the lab result as an upper bound on pure coding speed for a well-specified task. Use the field range as a more realistic expectation for team throughput under real constraints.
When someone quotes 55% as the business case for an engineering assistant, ask which workflow, which metric, and whether review and merge time were in the denominator.