ROI
also called Return on investment
Did it save more than it cost?
The value produced minus the cost of building and running it — where value is measured in time saved, tickets resolved, revenue, or risk reduced, and cost includes tokens, engineering and maintenance.
The most common failure is measuring the wrong thing: pilots that demo well and quietly consume tokens and maintenance with no workflow change. Attach the tool to a business metric before scaling it.
The honest version has a wide error bar, and it is worth saying so before someone produces a number. Attribution is genuinely hard, adoption is self-selected, and the measured effect depends on the task. What is defensible is a narrow claim about a specific workflow plus a direction of travel.
The measurement that had no control group
A team reports a large productivity gain after rolling out tooling.
The people who adopted it were the ones most motivated to make it work, which is a selection effect rather than a tool effect. Where a randomised comparison is not possible, say so and report the mechanism — time to first working version, review turnaround — rather than a headline percentage.
The value that was not developer speed
A team sees modest coding-time gains and a large drop in time to first useful answer.
Faster onboarding, quicker exploration of unfamiliar code and instant drafts of things nobody enjoyed writing are real returns. They do not show up in coding-time measurements, which is why narrow metrics miss where the value actually lands.
- paperThe impact of AI on developer productivity: evidence from GitHub Copilot ↗
A controlled experiment: developers with Copilot finished a specific JavaScript HTTP-server task 55.8% faster. One narrow task — do not generalise the number to software work.
- reportMeasuring the impact of early-2025 AI on experienced open-source developer productivity ↗
A randomised trial on experienced maintainers’ own repositories in which tasks took 19% longer with the AI tools, against participants’ expectation of a speed-up. A snapshot of one population and tool set.
- reportState of AI-assisted software development 2025 ↗
Frames AI as an amplifier of the organisation it lands in: the same tooling helps strong delivery cultures and compounds problems in weak ones.
Cost per Task
What one unit of real work actually costs.
Online Eval · Production evaluation
Measuring with real users, on real traffic.
Adoption Curve
The path from a few enthusiasts to real usage.
AI-Assisted Development
Day-to-day engineering with a model in the loop.