Token prices are easy to compare. Completed, accepted work is not. When a cheaper model needs more attempts, more human correction, or creates more waiting, the “savings” reverse. Price the whole path: inference, retries, review, and delay.
Define the unit as accepted work
Compare models on the same tasks: a support draft an agent accepts, a change that passes review, a summary a lead ships without rewriting. Counting generations alone favors the model that talks the most.
Hold the acceptance bar fixed. Otherwise you are not measuring cost—you are measuring ambition.
Add retries and review to the bill
A low per-token price with a 30% retry rate and heavy editing can cost more than a pricier model that lands cleanly once. Include retrieval, tooling, and the minutes a person spends verifying.
Delay is real cost when the workflow blocks someone. Overnight batch jobs and live customer replies do not share the same economics.
Route only after you have a baseline
Run a representative set on one stack. Record acceptance rate, latency, and total human minutes. Then test whether a cheaper model holds acceptance on a clearly defined subset.
Add fallbacks when you can detect hard cases. Avoid routing complexity that costs more to maintain than it saves.
Keep the decision reversible
Pricing and models change. Isolate provider-specific code and re-run the same evaluation set when you switch. The objective is dependable throughput at an acceptable total cost—not loyalty to a price sheet.