A vanity metric is any number that goes up regardless of whether anything good happened.
Token count is the purest example in AI: it climbs whether a model just solved a real problem or burned through its context window hallucinating a wrong answer. Neither outcome is distinguishable by looking at the number alone. "Tokenmaxxing Is Dead" made the case for why that matters. This is the follow-through — four measures that actually separate a working AI initiative from an expensive one, with the formula for each, specific enough to put directly into a dashboard, not just cite in a slide.
Cycle time saved
How much faster does a real process complete, end to end, with an agent in the loop versus without one — measured on the same process, not a different one that happens to look comparable.
Say a claims-triage process takes a reviewer 22 minutes end to end today: pull the claim, check three systems, decide, log the outcome. Route the same claim through an agent that does the lookups and drafts the decision for a human to confirm, and the reviewer's part drops to 4 minutes. Cycle time saved is 18 minutes — a real, bookable number, not a token count that says nothing about whether the claim got handled faster.
Error rate avoided
A before/after comparison too, but about quality, not speed: how many mistakes get caught before they ship, compared to the rate before the agent was in the loop.
Most process-improvement efforts skip measuring the "before" because it's unglamorous. It's also the entire point — an error rate you never measured isn't a number you're allowed to claim you improved.
Engineering hours reclaimed
The number executives actually understand, because it converts directly into "what would this team be doing instead."
An agent that needs constant babysitting reclaims almost nothing.
An agent that runs unattended reclaims close to the full hours-previously-spent figure. An agent that needs constant babysitting reclaims almost nothing — and that gap is exactly what a governance model with real HITL gates, not a rubber-stamp approval on everything, is supposed to protect. Approve everything and supervision time never drops. Gate only the steps that actually carry risk, and it does.
Cost per outcome
The one that actually replaces token count, because it's denominated the way procurement already thinks: cost per unit of work, not cost per unit of consumption.
"$0.03 per API call" means nothing on its own. "$1.40 to fully process one invoice, end to end, compared to the $6.20 it cost a person to do the same thing manually" — that's a number a CFO can put directly into a business case.
What this actually takes
None of these four numbers show up automatically. They require three things working together, in this order:
- A real baseline, captured before the agent touched the process — not reconstructed from memory afterward.
- Attribution that ties a specific cost to a specific workflow instance, not a monthly aggregate that arrives too late to act on.
- A system that keeps the "before" number around long enough to actually compare it against the "after," instead of living in someone's spreadsheet for exactly one board meeting.
That's what routing that attributes cost per request, and a dashboard built for exactly this comparison, are for — not a nice-to-have, the actual precondition for measuring any of this honestly.
Where this leaves you
Token count will keep climbing no matter which of these four numbers move. That's fine — it was never the metric that mattered.
Track these four instead, and the business case for AI stops being an argument and starts being an invoice.