Everyone measuring an AI workflow starts with speed because speed is easy to capture and flattering to report. Tickets per hour, words per minute, tasks completed per day — all trivial to log, all trending up after you plug in an agent. None of them tell you whether the output was right. A system that answers faster and wrong is not an improvement. It is a liability with better marketing.
Why speed is the metric everyone reaches for
Speed is easy to measure and easy to show a client, which is exactly why it gets reported even when it is not the thing that matters. A timestamp requires no judgment. Quality requires someone to actually read the output, compare it to a standard, and decide if it is good — which is slower, more subjective, and less fun to put in a slide. So teams default to the number that is cheap to produce. The result is a dashboard that looks like progress while the actual cost of the system, measured in rework and customer damage, sits outside the frame entirely.
This matters more with agentic systems than with the automation that came before them, because agents make judgment calls instead of following fixed rules. A script that fails, fails the same way every time and gets caught in testing. An agent that misjudges a customer’s intent, drafts a confident wrong answer, and sends it, fails silently and differently each time. Speed metrics cannot see that failure mode at all.
What a faster wrong answer actually costs
A wrong answer delivered quickly does not save time, it moves the time cost downstream and usually multiplies it. Someone has to notice the error, trace it back, explain what went wrong, and redo the work — and by then the error may have already reached a customer, a filing, or a dependent process. The thirty seconds saved at generation time can cost an hour at the correction stage, plus whatever trust was spent in between. This is the honest maths that automation pitches tend to skip: the saving is real at the point of output and imaginary once you follow the error through the system.
The fix is not to slow the system down on principle. It is to stop treating speed as a proxy for value and start measuring the thing you actually care about, which is whether the work was correct the first time.
Three numbers worth tracking
Error rate is the obvious one and the one people most often skip measuring directly, instead inferring it from complaints or support tickets after the fact. Measure it at the point of output, not at the point of complaint — sample a fixed percentage of outputs and have a human check them against a defined standard, every cycle, not just when something feels off.
Rework rate tells you how much of the system’s output needs to be touched again before it is usable. This is different from error rate because small errors that get caught early are cheap, and the same error caught late is not. Track how far downstream a correction happens, not just whether one happened.
Acceptance rate without edits is the cleanest single number, because it combines the other two into something a non-technical stakeholder can understand. If a human has to change the AI’s output before using it, that is not a pass, even if the change was small. Report the percentage accepted as-is, and watch it over time rather than trusting a single week’s sample — AI output quality is not stable, and one good week proves little.
Building this into a workflow instead of an audit
The measurement has to be built into the workflow itself, or it will not happen consistently, because nobody keeps a manual quality log going for more than a few weeks. This is the same discipline covered in the workflows library on this site: the check is a step in the process, not a report generated afterward. A lightweight version is a sampling rule — every twentieth output gets a human review before it ships, logged against the same three numbers above, reviewed weekly.
Where this breaks down is scale. One person checking every twentieth output works for a team running a few hundred tasks a week. It does not work at ten thousand, and that is usually the point where teams quietly stop checking at all and let the speed metric take over by default, because nobody budgeted time for quality review at the new volume. If you are scaling an agentic system past the point where manual sampling is realistic, that is worth planning for before the volume arrives, not after the first bad quarter — which is the kind of problem the services side of this work tends to get called in for.
Who should not chase speed at all
Any process with an asymmetric cost of error — legal, financial, medical, anything customer-facing with a refund or reputation risk attached — should not optimize for speed until quality is measured and stable. The downside of a slow correct answer in these domains is bounded. The downside of a fast wrong one is not. Optimize for speed only once you know your error rate, and only up to the point where that rate stays flat. Past that point, more speed is just more errors, faster.