"What's the ROI of AI?" is the wrong question right now.

Introduction

Google Cloud recently published their 2026 ROI of AI-Assisted Software Development framework. Branded as a DORA report. Reaction inside the DevOps community came fast. Daniel Terhorst-North, originator of BDD and longtime DevOps figure, wrote on LinkedIn:

This isn't a #DORA report, it's just a Google puff piece with 'DORA' written above it in crayon.

Daniel Terhorst-North

He's right. The 2026 piece is a Google Cloud consulting framework with the DORA brand wrapper. Under Nicole Forsgren's tenure, DORA reports were defended by methodology sections, statistical disclosures, and credible-interval charts. The 2026 ROI piece has a J-Curve and an "Instability Tax" calculator. As Dan put it: "The Research and Assessment parts of DORA seem to have evaporated altogether." The actual research-backed work, the DORA 2025 State of AI-Assisted Software Development report, is more honest about what's measured and what isn't. It's a survey of practitioner experience. It doesn't claim to compute your ROI for you.

Around the same time, MIT Project NANDA published State of AI in Business 2025, the now-famous "95% report." Despite $30-40 billion in enterprise GenAI investment, 95% of organizations show no measurable P&L impact. NANDA's own diagnosis:

The core barrier to scaling is not infrastructure, regulation, or talent. It is learning.

State of AI in Business 2025

I'd been trying to figure out how to instrument AI ROI for a while before either of these landed. The Google framework arriving, and getting properly critiqued, is what convinced me to write this down. We need the right metric and the right method. We need to be honest about what's lever, what's gauge, and what's marketing.

This post lays out a working framework, the practical metric to instrument first, and a DMAIC walkthrough for getting from "we have no instrumentation" to "we measure what we can move."

TL;DR

Three claims:

1. ROI is a lagging gauge. You can't manage it directly. Manage what's upstream.
2. Tokens fail Bezos's causality test. They are an input metric in name only.
3. Cost per output is the right V1 input metric. The mature version pulls outcome validation into the ratio.

The rest is detail and receipts.

Definitions: getting everyone on the same page

Two complementary frameworks. They say the same thing at different levels of granularity.

Working Backwards

The book Working Backwards makes a binary distinction:

• Input metrics are levers you can move. Selection, price, in-stock availability, fast shipping were Bezos's canonical four for Amazon retail.
• Output metrics are gauges that lag. Revenue, profit, customer count, stock price.

The principle compresses to one line: Input metrics are levers; output metrics are gauges. You manage by pulling levers, not by staring at gauges.

Output Metrics Show Results. Input Metrics Provide Guidance.

Working Backwards

Bezos demanded two properties of an input metric:

1. Controllable. The team can move it with effort.
2. Causally tied. Pulling the lever empirically moves the gauge.

Selection wasn't arbitrary. Amazon validated empirically that more SKUs caused more purchases. If pulling the lever doesn't move the gauge, it's a vanity metric or the wrong choice, not an input metric. (We'll come back to this point when we talk about tokens.)

Impact Mapping

Impact Mapping uses a more granular five-tier model. Each tier names something specific that organizations actually argue about:

• Resources: what we expend. Employees, time, money, tokens, attention.
• Activities: the work people (and agents) do. Coding, reviewing, testing, deploying, agentic workflows.
• Outputs: what we build. A new feature, a merged PR, a deploy, an evidence artifact.
• Outcomes: what people do with the outputs. Adoption, retention, satisfaction, behavior change.
• Impacts: what result the outcomes generate. ROI, customer success, audit posture, mission outcomes.

The chain runs left to right: Resources → Activities → Outputs → Outcomes → Impacts.

You don't get outcomes by managing outcomes. You manage the resources, activities, and outputs that cause outcomes, which compound into impacts. That's the operationalized causality principle. Impact Mapping is explicit about this: focus on what needs to change, for whom, and why, so you don't fall into the feature factory trap.

In practice, different parts of an organization focus on different tiers. Execution teams focus on Resources and Activities (the work they do, the inputs they consume). Engineering managers and PMs often own Outputs (what gets shipped). Product strategy leaders own Outcomes (what users actually do). Senior leaders and boards focus on Impacts (the lagging mission and business gauges). The whole chain has to hold for the impact gauge to move.

How the two frameworks map

Both frameworks describe the same shape of the world. Working Backwards uses a binary; Impact Mapping uses a quintet. The mapping:

Working Backwards collapses the five Impact Mapping tiers into two: controllable (input metrics, our levers) and lagging (output metrics, our gauges). Impact Mapping expands the same idea into five tiers with explicit causality.

Both are correct. Use Impact Mapping when you need precision about which layer you're talking about. Use Working Backwards when you need the simpler binary (am I managing a lever or staring at a gauge?).

For the rest of this post, when I say "lever" I mean resources, activities, or outputs. When I say "gauge" I mean outcomes or impacts.

Why "What's the ROI?" goes nowhere

Restate the framework against the question: ROI is impact. Impact is the gauge. The gauge lags.

Saying "what's the ROI?" is asking "where does the gauge sit right now?" Sometimes a useful question. Almost never an actionable one, because gauges don't respond to direct management. They respond to upstream changes that compound.

The MIT NANDA report puts a number on this. Despite $30-40 billion in enterprise GenAI investment, 95% of organizations report no measurable P&L impact. The 5% who do report it didn't get there by managing the gauge. They got there by being deliberate about the levers underneath. NANDA's diagnosis is worth repeating: "The core barrier to scaling is not infrastructure, regulation, or talent. It is learning."

That diagnosis matters. Learning is itself a lever. It's the meta-activity that lets you discover which other levers actually drive the gauge in your context. The 5% who succeed are running the learning loop: instrument an input metric, watch what moves, refine, instrument again. The 95% are picking a metric (often token spend), staring at it, and waiting for ROI to follow. It doesn't.

The 95% are managing gauges. The 5% are managing levers, and learning which levers to manage.

Why tokens are not the right input metric

Once you accept "manage levers, not gauges," the next instinct in 2026 is to count tokens. We burned 50 million tokens last week. The top decile of token-using engineers ships more code. Therefore tokens are the lever.

Tokens fail Bezos's causality test.

Nicholas Arcolano (Jellyfish, Q1 2026) measured ~7,500 engineers and found:

• The top decile of token usage burns roughly 10x the tokens of the median.
• For roughly 2x the PRs.

10x the tokens for 2x the PRs. Pulling harder on the token lever doesn't proportionally move the output gauge.

Tokens are controllable and leading (which makes them a candidate input metric in the WB sense), but they're not causally tied to the output gauge at scale. WB's definition of a wrong input metric is exactly this: easy to measure, easy to game, doesn't actually drive the thing you care about.

Matt Pocock (May 2026) made the same point from a different angle. Comparing markdown and HTML output formats in terms of token spend, he wrote, is "comparing apples with cricket bats." The right metric for that comparison is value per token: same critique of input-counting, different anchor (denominator stays at tokens, but the numerator becomes value). His resolution works for format-comparison decisions where token spend is the comparison axis. The cost-per-output resolution works for finops decisions where the dollar denominator is what leadership tracks.

Both reject the "more tokens = more progress" reading. Both move the metric to something that has to relate to value, not just quantity.

Token spend is a useful diagnostic. It tells you something is happening. It is not a target.

The right starting metric: cost per output

For a finops conversation about AI in an engineering org, the right V1 input metric is cost per output.

An output is what your team produces: a merged PR, a deploy, an evidence artifact for an audit, a shipped feature. Your team can move cost per output. It's leading. And, if the outputs you're producing are actually tied to outcomes and impacts, it's causally meaningful.

That conditional matters more than the metric itself. Let me explain.

The feature factory risk

Cost per output instruments the cost side. It does not validate that the outputs are the right ones.

You can ship 100 PRs at a great cost per output and have zero impact on outcomes or impacts. That's the feature factory: a team that's efficient at producing outputs that don't move the gauges.

Bezos's "selection, price, delivery speed" weren't arbitrary output counts. They were validated levers. Amazon ran the experiments. More selection causally drove more purchases. Lower price drove more purchases. Faster shipping drove more retention. Each of the four input metrics earned its status by passing the causality test against revenue.

Cost per output inherits this lesson with a caveat: it's the cost-efficiency input metric. It assumes your outputs actually ladder to outcomes. The validation work (making sure the outputs you're shipping cause the outcomes you want) is a separate discipline that has to happen alongside the cost instrumentation.

If you have great cost-per-output and no validation, you have an efficient feature factory. That's not a win.

V1 → V∞: the iteration ladder

Finding the right input metric is iterative and never done. Amazon's selection metric went through four versions. Same logic applies here.

Cost per output is V1, the starting point most orgs in 2026 can actually instrument. The mature versions pull outcome causality progressively into the ratio:

The catch: V2 and beyond require outcome-tracking infrastructure most orgs don't have in 2026. Adoption analytics, feature usage telemetry, attribution from features to whatever your impact unit actually is (revenue, retention, mission readiness, audit posture): these are real engineering investments. Most enterprises haven't made them.

So the practical move is V1 first. Get the instrumentation in place. Then iterate toward V2.

This is also where learning-as-a-lever shows up operationally. Each version is a hypothesis about which input metric drives the gauge. You instrument it. You watch what happens. If pulling the lever moves the gauge, the metric earns its place. If not, you refine. The MIT NANDA finding is the macro version of this: 95% of orgs don't run this loop at all, so they never find out which levers matter.

A multi-metric scorecard is the honest answer at scale

Bezos didn't pick one input metric. Amazon used four (selection, price, in-stock, speed). For engineering AI work, the parallel basket is something like:

• Cost efficiency: cost per output
• Process efficiency: lead time / cycle time per output (DORA-style)
• Outcome causality: adoption rate of shipped outputs
• Quality: change failure rate / rollback rate

Cost per output is one metric in the basket, not the whole basket. The post simplifies to one for rhetorical clarity. In practice, the mature org tracks all four (and probably more), and the rituals around them resemble Amazon's Weekly Business Review.

V1 first. Then iterate.

DMAIC: how to actually get there

Cost per output is the metric. DMAIC is the method. Six Sigma's framework (Define, Measure, Analyze, Improve, Control) gives you a sequence for moving from "we don't measure this" to "we measure it consistently and act on what we see."

Define

Two things to name:

1. The output. What is your team actually producing? Be specific. "PRs" is one answer. "Shipped features in production" is another. "Compliance evidence artifacts" is another. The choice matters because it determines what the cost-per-output ratio measures. PRs include work that gets reverted. Shipped features include work that doesn't get used. Evidence artifacts include work that didn't change anything for the customer. Choose deliberately.

2. The impact it ladders to. Which outcomes (behavioral changes) do these outputs drive? Which mission or business impact compounds from those outcomes? This is the value-to-impact conversation, and most orgs skip it.

If you skip it, cost per output just powers a feature factory.

The Atlassian 2026 State of Teams survey reports the gap directly: 73% of executives measure AI success through productivity, while 46% admit organizational AI objectives are unclear. Two angles on the same gap. You can't measure what you haven't named.

The output of Define is a written, agreed answer to two questions: "What are we measuring?" and "Why does it matter?" If your team can't answer both in one paragraph, you're not done with Define. Skipping ahead to Measure won't help.

Measure

This is where most orgs are stuck. It's partly a tool problem.

Subscriptions hide the meter. Cursor, Claude Code Max, Copilot. Flat price, no per-task visibility. You can't compute cost per output from a flat $200/month subscription. This is the canonical 2026 instrumentation problem.

API keys help, partially. They stop at whatever scope you set: project, team, developer. They don't reach task-level or PR-level granularity without additional work. You can attribute spend to a project; you can't attribute it to a specific feature shipped.

No unified view. A real cost report stitches across Anthropic, OpenAI, Cursor, Copilot, the corporate card, and any infrastructure spend. By hand. Weekly. Nobody does it consistently.

Finance certifies the reconciliation, not Tech. Structural independence keeps the data unbiased. Engineering teams have an incentive to present favorable numbers. Finance doesn't. The team that owns the metric should not be the team that certifies it.

Hidden tokens and silent rate changes. Reasoning tokens get billed without showing in the UI. Tokenizers change between releases. Cache behavior shifts. Region pricing premiums apply silently (Anthropic charges 1.1× for US-region inference per Ramp's spend pipeline catalog). The meter is technically running; the gauge is built so you can't read it.

The instrumentation gap is real. Vendors haven't yet built the per-task visibility most orgs need. That's partly a vendor pressure problem. Until enough customers demand the meter, vendors will keep selling subscriptions and vague practices that obscure it.

Analyze

Once you can see cost per output, ask why the distribution looks the way it does:

• Why is one developer 3x the median cost per PR?
• Why is one project 5x another's cost per output?
• Why does one team's cost per shipped feature compound while another's flatlines?

Five Whys, not first explanation. The first explanation is almost always wrong. The actual root cause usually sits two or three levels deeper.

Ramp's leak catalog (from their AI cost-management work) is a useful starting checklist of common causes:

• Oversized models for the task
• No prompt or response caching
• Runaway agent loops
• Abandoned experiments still running in the background
• Prompt bloat (no context compression, no compaction)

Each of these has a different remediation, and conflating them is how Improve cycles burn budget without moving the metric.

Data without anecdotes is incomplete; anecdotes without data are unreliable. The strongest Analyze sessions pair the cost-per-output distribution with stories from the engineers at the tails. Why is Sarah's cost per PR 3x the median? Sometimes she's tackling harder problems. Sometimes she's stuck in a runaway loop. Sometimes her CLAUDE.md is missing context that the rest of the team has. The data tells you where to look. The anecdotes tell you what you're looking at.

Improve

Once you've identified what's driving cost variance, intervene. Common moves:

• Reclaim idle subscription seats. Most subscriptions have idle users at any given month. The data is there if you can see it.
• Standardize CLAUDE.md and AGENTS.md context across the org. Inconsistent context makes prompts inefficient.
• Right-size models by verification cost, not by vibes. Most tasks don't need Opus. The shift to a smaller model often pays for itself in days.
• Cache aggressively. Prompt caching, response caching, KV caching where the model supports it.

The discipline: only move to Improve after Measure works. Skipping DMAIC steps causes thrash. Improving what you can't measure means improving what you can't see results from. That produces motion without progress.

Control

Make the metric persistent.

• Monthly utilization review.
• Idle-seat alerts at the team level.
• Weekly cost-per-output on the Tech WBR (Weekly Business Review).

The WBR is itself a Working Backwards mechanism, not just a meeting. Its value comes from consistent format, exception focus, metric ownership, and the discipline of doing it every single week. Skipping a week breaks the loop. Letting metric ownership drift breaks the loop. Letting the format change every week breaks the loop.

Discipline is the price of the metric being persistent.

Why Bezos didn't track ROI

Bezos didn't track ROI of free shipping at Amazon. He tracked input metrics: selection, price, and delivery speed. ROI compounded as a consequence.

This is the canonical example. Input metrics are levers. Output metrics are gauges. You manage by pulling levers. If your team only reviews revenue and profit, you're looking in the rearview mirror.

Here's the part that helps me explain it. Imagine you want to finish a marathon under four hours. The race time is the impact. You can stare at a calendar and worry about it. The calendar doesn't run faster.

What does move? The activities that compound into capability: weekly mileage, long runs, tempo workouts, recovery days, sleep, nutrition. You map them all out. You stick to them. Your fitness improves. Race-day capability follows. If you skip the activities and just stare at the goal, you don't get to the goal.

Now add a coach. Now add a relay team going for a four-by-marathon record. The plan still matters. The coordination is harder. The number of moving parts goes up. The principle holds: you don't manage the time. You manage the activities that produce the capability that produces the time.

Notice that race time isn't a dollar figure. The unit of impact in your org might not be a dollar figure either. The ladder works the same way regardless of what your denominator actually is. Revenue dollars for a commercial team, mission readiness for a federal team, accreditation cycles saved for a compliance team. The shape of the work is the same; the currency changes.

That is exactly Bezos's argument applied to athletic training. Selection, price, and delivery speed were Amazon's inputs. They compounded into customer behavior (outcomes), which compounded into revenue and customer count (impacts). Bezos managed the inputs. The impact followed. The translation for AI:

• Impact is the gauge (revenue, retention, mission readiness, audit posture, whatever your org actually measures). You can't move it directly.
• Cost per output is the lever you can move this week.
• Outcomes (adoption, retention, sustained usage) are the validation that the outputs you're shipping actually ladder to impact.
• Impact compounds when the lever moves and the chain holds.

If the lever moves and impact doesn't follow, you have a feature factory. The Define step is where you catch that, by naming the output and the impact it should ladder to, before you're optimizing the cost of producing things that don't matter.

The next move isn't a decision. It's instrumentation.

Most engineering leaders in 2026 are stuck between Define and Measure. The conversation about whether AI is worth it is dangerous because vibes are not data. Stop debating whether AI is worth it. Start measuring cost per output per project. ROI answers itself once the levers are visible.

The next move isn't a strategic decision. It's instrumentation. And, partly, vendor pressure to expose the meter that vendors currently obscure.

What input metrics is your team actually tracking? Whose budget do they live on: org, department, project, or person?

See the associated LinkedIn post.