I spent a while optimising for value per token. It was better than what most teams measure, and it was still wrong. That correction is the actual subject of this piece.
The constraint is physical
No invoice ever surprised us. What started this was a debate that has nothing to do with product management: the supply and demand of tokens.
Nobody argues about demand. It rises, and with agentic systems it rises in steps rather than curves. Gartner puts the consumption of an agentic workflow at five to thirty times a single chat request. The reason is unglamorous and therefore easy to miss. On every loop iteration, an agent resends the full accumulated context. After twenty steps you have paid for the original context twenty times.
Supply is the more interesting side, because it is not negotiable. A token is compute, compute is chips, chips run on electricity. Elon Musk put it briefly to The Economist on 23 July 2026: "the constraints on AI are essentially electricity and AI chips". He added a geography that gets discussed less often. Outside China, electrical power is the bottleneck, because chips come off the line faster than power plants come online. In China it runs the other way: energy is plentiful, high-end chips are not. Though Chinese manufacturers are closing that gap.
The International Energy Agency sized this in April 2025 in Energy and AI. Data centre electricity consumption rises from 415 terawatt hours in 2024 to roughly 945 terawatt hours by 2030, with AI-optimised facilities growing around 30 percent a year.
Honesty requires the counter-calculation. Carbon Brief pointed out that this increase accounts for only about eight percent of total projected demand growth, less than electric vehicles or air conditioning. So the bottleneck is not global. It is regionally concentrated, which makes it no smaller for product decisions, only harder to forecast.
The window, and the trap inside it
At the BlackRock US Infrastructure Summit in March 2026, Sam Altman said intelligence would become "a utility like electricity or water", metered. The clause that followed matters more: if there is not enough, either you cannot sell it or the price goes up sharply.
The claim that tokens are broadly sold below cost does not hold in that generality. On the API layer, providers are closer to profitability than the public debate suggests. What is documented is more specific: unlimited agent flat rates and consumer subscriptions were subsidised, and that model is being dismantled right now. In the first quarter of 2026 the major providers moved enterprise customers to token-based billing. Uber burned its 2026 AI budget in four months.
That opens a window worth using, but not the way most teams are using it. Put your entire development pipeline on subsidised token prices and you have built a dependency whose price you do not know. The better play is the opposite: use the window to get existing products into shape and to prove real results. Under one premise we measure decisions against: as little complexity as necessary, as much margin per completed task as possible.
Which is where I got the arithmetic wrong for a while.
The metric I had wrong
Start with what most teams measure: cost per token.
That number appears on the invoice, so it gets reported. As a steering metric it is worthless, because it only knows the buying side. A team optimising cost per token trims context, picks smaller models and caps output length, without knowing whether anything a customer cares about got cheaper.
My own attempt to fix that was value per token. At least it puts benefit in the numerator. It is a better metric. It is still wrong, for two reasons.
The first is a counterexample from our own work. You improve an agent's instructions so it receives more context and therefore gets the right answer on the first attempt more often. Token consumption per attempt goes up. Failed runs drop sharply. Your economics improve substantially. And your value per token falls.
The metric points the wrong way while you are doing the right thing. That is not an edge case. It happens on every improvement to hit rate.
The second reason is simpler. Value per token is blind to price. If token prices halve tomorrow, the number does not move, even though your economics just doubled. A metric that fails to register a price halving is not an economic metric. It measures technical efficiency.
The error sits in the denominator. A token is an input, not a unit of work delivered. You need a unit in which work actually accrues.
Matei Zaharia of Databricks compressed that step for The Register in July 2026: "Cheaper per-token does not imply cheaper per-task."
So the unit is the completed task. An allocated line item. A cleared goods receipt. A posted document. Something that is either finished or not.
Now put value per task against cost per task and the tasks cancel out. What remains is value over cost, a dimensionless ratio with a natural decision threshold at one. It compares across entirely different use cases and it responds correctly to price changes. Everything value per token fails at.
The fact that the task cancels does not make it redundant. It forces you to define the smallest unit of completed work, and that is the real thinking. The algebra is trivial. Deciding what counts as a completed task in your product is not.
Two problems remain.
Failed runs disappear. If you average cost per task across all attempts, your hit rate has vanished into the number. The honest denominator is every run started, including retries, corrections and human rework. The numerator counts only the successful ones.
Scale gets lost. A task worth ten at a cost of one gives a ratio of ten. A task worth ten thousand at a cost of two thousand gives five. Optimise the ratio alone and you pick the first and leave the second on the table. That is the old conflict between return and absolute contribution.
Which brings me to what we use now: contribution margin per completed task. The value of the finished transaction, minus every variable cost incurred to complete it, including failed runs, tool calls and human review. Multiplied by volume, that gives you absolute contribution.
This solves all four problems at once. The denominator is honest, the threshold sits at zero, failed runs are automatically in there as variable cost with no revenue, and multiplying by volume restores scale.
The real gain is elsewhere. Contribution margin is not an AI metric. It is the metric finance people have used for decades. A board member, a CFO, an owner in the mid-market needs no translation and no patience for another new piece of vocabulary. Translating a new problem into an established language works better than teaching the organisation one more number.
What that means in practice
Before we build an AI feature, we do not design the architecture. We design the experiment. Three questions, in this order. What counts as a completed task here? What is it worth to the customer? What does it cost us to complete, failed runs included?
Implementation starts once those have answers.
This sounds like added bureaucracy and does the opposite. It forces product management to know the contribution margin of every use case before the use case exists. Without that number you cannot build a pricing model in a volatile cost environment that survives a model migration. Nor can you decide which feature to cut when a provider adjusts prices.
The method is unremarkable and comes from three directions: user research for what counts as value, technical system modelling for the cost side, and commercial judgement for what a customer will pay.
The mechanism behind a 25x reduction
The cost side of that equation can be worked hard. The lever is decomposition.
An agent's capabilities have to break into small pieces. This is not a software architecture preference, it is a requirement of the domain. In enterprise software, every agent capability sits on top of a business transaction with a defined correct outcome. A line item is allocated correctly or it is not. A document is complete or it is not. The error tolerance of professional users who work in the system eight hours a day, and whose own output gets audited, is close to zero. And when something goes wrong, you need to know which step failed, not that the agent failed.
That is precisely why targeted evaluations can only be written for small, clearly bounded capabilities. Those evaluations first secure the quality of each individual capability. Their second benefit is economic: they are the instrument that determines the cheapest sufficient model for production.
The size of this lever is documented. RouteLLM, work out of Berkeley presented at ICLR 2025, achieved around 85 percent cost reduction at roughly 95 percent of the strong model's quality, by routing only a small share of requests to the expensive model.
On one of our agentic product features we cut production token cost by a factor of 25 with no quality loss. Because the hit rate held, that reduction flows straight through to contribution margin. It came from no trick. The quality gates were in place early enough to turn model selection into a measurement question rather than a matter of taste.
One detail is an observation from our own practice, not a published finding. The judge model in your evaluation should always be a frontier model, even when a smaller one runs in production. The reason is not judgement quality alone. A frontier judge moves your reference point along with the state of the art. Without it you measure your quality against a picture that quietly ages, and perceived user value erodes over months without anyone noticing. The evaluation still reports green.
The weaknesses of this approach are well documented and worth knowing before you rely on it. Judge models favour longer answers, they favour whichever option they see first, and they favour output resembling their own. Randomise order, normalise for length, average across runs.
What cost more than expected
Not the models. The tools.
Tool calls are the underestimated cost driver, and Model Context Protocol implementations are where the largest optimisation potential sits. In November 2025 Anthropic showed a workflow that previously consumed around 150,000 tokens dropping to roughly 2,000 when tool calls were replaced by code execution. The reason is mundane: a setup with five servers and around sixty tools burns roughly 55,000 tokens before the model has read the actual question.
User behaviour was the pleasant surprise. In professional environments, users generate unplanned expensive consumption patterns less often than consumers do. The scope of use is narrower and more purposeful. It cannot be ruled out, which is why we hold one rule without exception: consumption monitoring is not a feature you add after launch. It has to be in place before. Without it you do not know the denominator of your own equation.
On context length the finding splits in two, and the split surprised me. In the high-value ERP use cases we work on, context length is currently not the limiting factor. The transaction is bounded and so is the relevant data. Where it genuinely does bind is internal use in software engineering, as soon as monolithic components are involved. There it remains a hard limit, one we only get past because our experienced engineering teams can judge which slice of a system built over decades is actually needed for a given task. That judgement has no substitute yet.
The objection from inside
It did not arrive as "models will get cheaper anyway". It arrived in the harder form, the one that is legitimate.
Launch dates and broad go-to-market campaigns create delivery pressure in product development. That pressure is legitimate, and AI has raised it rather than lowered it. When development speed multiplies, expectations move with it. Anyone who then asks for a measurement design before every feature looks like the person applying the brakes.
My answer is not to dispute the pressure. It is that only agile product development can meet those expectations at all, and that economic measurement is part of that discipline rather than its opposite. A feature whose contribution margin you do not know has not shipped fast. It has shipped on credit.
And the hope that prices will fall? Moonshot AI's current flagship is the case study. Kimi K3 launched in July 2026, sits in the top group of independent comparisons, and costs about what mid-tier Western models cost per token. It also produces roughly twice the output tokens of the median comparable model across a full benchmark run, according to Artificial Analysis. The lower price disappears into verbosity. Exactly the effect Zaharia named, and exactly why the denominator has to be the task rather than the token. Until you measure contribution margin per completed task, you cannot detect the saving from a model switch. You can only assert it.
And the objection from outside
Which brings me to the real argument, one the industry appears to have settled and, in my view, settled wrongly.
In December 2024 on the BG2 podcast, Satya Nadella framed the thesis that now shows up in every second deck: "the notion that business applications exist, that's probably where they'll all collapse". Business applications, on this view, are databases with some logic on top, and the logic migrates into the AI layer. Foundation Capital turned this into the term service-as-software and a trillion dollar market narrative.
I think that thesis is wrong, at least for the systems I am describing here.
An ERP or accounting system refined over decades until it satisfies the most exacting auditor to the last decimal place will not be displaced by a non-deterministic technology. This is not technical conservatism. It is the requirements.
Germany happens to have written those requirements down more explicitly than most jurisdictions. The GoBD, the principles governing proper bookkeeping in electronic form, demand traceability, immutability and machine auditability. Every transaction must remain fully traceable, and a change must not obscure what was there before. Other jurisdictions have their own version of this. Germany's is simply the sharpest instance, which makes it a useful test case for anyone building AI into a system of record anywhere.
That non-determinism is not a fringe concern was set out precisely by Thinking Machines Lab in September 2025 in Defeating Nondeterminism in LLM Inference. The main cause of varying output is not randomness in the model but a lack of batch invariance. Batch size shifts with server load, the order of floating point accumulation changes, and so does the result. Even at temperature zero.
In an ERP context that leads to constraints other products can skip. Customer data must not be used for training by processors. The scope of use is narrow enough that temperature can be kept minimal, which maximises reproducibility without letting anyone claim full determinism. And decisive information is presented to users in validated form rather than generated. In our domain that distinction is the difference.
The empirical picture supports the scepticism, with recent numbers. Uber spent its 2026 AI budget in four months. Salesforce changed the pricing of its agent platform three times in eighteen months. Neither is a technology problem. Both are calculation problems that only surfaced in production. Gartner expects more than 40 percent of agentic AI projects to be cancelled by the end of 2027, citing escalating costs and unclear value.
My counter-thesis is not that AI has no place in these systems. It is that the direction runs the other way. The rigid process models of an established ERP are what allow fast digitalisation initiatives to keep their quality. Without formalised process quality, the gains from hastily implemented automation return as process errors, at a multiple of the cost saved. This has nothing to do with looking backwards. It is the same calculation as the task, one level up.
Where I might be wrong
Three places, and the first is already in this text.
I argued for a metric I now consider insufficient. It is entirely possible that contribution margin per completed task is also an intermediate step. The obvious weak point is the numerator. What a completed task is worth to the customer can only be estimated, and that estimate is more contestable than any cost line. Anyone with a better method for valuing the benefit side has the next move.
Second: if batch-invariant inference becomes standard and reproducible output becomes the norm, my determinism argument loses part of its edge. Some serving systems have already adopted the approach. The question then shifts from reproducibility to auditability, which is harder to answer.
Third: my measurement comes from a domain with tightly bounded transactions and professional users. Whether a factor of 25 is reachable in more open-ended use cases, I do not know. My guess is that it is smaller there.
What I want to hear is the counter-position from someone who runs the numbers differently. Who measures contribution margin per completed task in their own organisation and reaches a different conclusion than I do?
Sources and dates: Musk in conversation with The Economist, 23 July 2026. IEA, Energy and AI, April 2025. Altman, BlackRock US Infrastructure Summit, March 2026. Zaharia via The Register, 13 July 2026. Artificial Analysis on Kimi K3, July 2026. Uber and Salesforce figures via Forbes, July 2026. Nadella, BG2 podcast, December 2024. Thinking Machines Lab, September 2025. Anthropic Engineering on code execution with MCP, November 2025. Gartner press release on agentic projects, 25 June 2025. RouteLLM, ICLR 2025. Model pricing and consumption data move quickly; figures as of August 2026.