The price of a million tokens is easy to compare. The cost of a completed, accepted task is more useful—and harder to measure. That gap is where teams overspend on “cheap” models and underspend on ones that finish the job.
Choose the unit of value
Define the outcome before comparing providers. For support, it might be a correctly prepared response an agent accepts. For engineering, it might be a change that passes review and the relevant checks.
Keep that definition stable. Counting generated drafts for one model and accepted work for another makes the comparison meaningless.
Include the work around the model
A useful cost estimate includes inference, retrieval, retries, infrastructure, and human review. If one model needs more attempts or more correction, a lower API bill can hide a more expensive workflow.
Latency changes the economics. A workflow that blocks a person should be assessed differently from a batch job that runs overnight.
Route only after you have a baseline
Start with a representative set of tasks and a simple implementation. Record acceptance rate, latency, and total effort. Then test whether a less expensive model can handle a clearly defined subset without lowering acceptance.
Add a fallback when the system can reliably identify cases that need it. Avoid a routing setup that costs more to maintain than it saves.
Keep the decision reversible
Model behavior, pricing, and your workload will change. Keep evaluations close to the application and isolate provider-specific code. Re-run the same tasks when you make a meaningful change.
The objective is a dependable workflow at an acceptable cost. The provider and model are implementation choices in service of that objective.