Agent pilots rarely fail because the model was wrong. They fail because nobody built the scaffolding around the model: logging that attributes errors to a step, an owner who answers when it breaks, and a cost model set before launch instead of discovered after the invoice arrives. The demo worked. Production didn’t. Those are different products, and treating them as the same one is the first mistake.
The demo-to-production gap
A demo succeeds because someone curated its inputs, and production fails because nobody curates anything. In a demo you type three clean prompts you already know will work. In production, the input is a customer’s badly-formatted email, a PDF with a scanned signature block, or a database record with a null field the schema didn’t account for. The agent that handled the demo flawlessly has never seen any of that.
The fix is not a better model. It’s running the pilot against a sample of real, ugly production data before anyone calls it a pilot. If you can’t get real data, get the second-worst data you have — the stuff support tickets are made of, not the stuff in the pitch deck. Teams that skip this step find out about the gap from a customer, which is the most expensive way to find out about anything.
Unattributed failure
When an agent chain fails, the question that matters is which step failed and why, and most pilots cannot answer it. The output was wrong, or the task stalled, and the logs show a wall of tokens with no marker for where the reasoning went sideways. Was it a bad retrieval, a tool call that returned an error the agent silently swallowed, or a prompt that drifted after step four?
Without attribution, every failure gets the same fix: rewrite the prompt and hope. That’s not debugging, it’s guessing with extra steps. The teams whose pilots survive contact instrument each step separately — input, tool call, output, confidence — before the agent touches a real user. This is the same discipline as instrumenting a distributed system, and agent chains are distributed systems whether anyone planned them that way or not. If you’re building this from scratch, it’s worth looking at how a structured workflow separates these stages explicitly rather than letting the model own the whole chain as one black box.
No owner
A pilot survives its first month because someone built it. It survives its sixth month only if someone owns it, and “owns it” means something specific: gets paged when it breaks, has authority to change the prompt or the pipeline, and reports on its failure rate to someone who cares. Most pilots have a builder, not an owner. The builder moves to the next project, the pilot keeps running unattended, and the first anyone notices a problem is when a customer complains or a bill spikes.
This is an organizational failure dressed up as a technical one. The fix is boring: name the owner before launch, not after the first incident, and give them a standing slot to report status. If nobody in the room is willing to take that job, that’s useful information too — it means the pilot isn’t important enough to run in production yet, whatever the demo suggested.
Cost surprise
Token cost scales with usage in a way that a ten-user pilot completely hides. An agent that costs a few cents per run in testing can cost multiples of that in production once retries, tool calls, and longer context windows from real documents get involved. Nobody modelled it because nobody ran the arithmetic past the number of testers in the room.
Set a cost ceiling before launch, not after the first bill. Know the cost per successful task, not per API call — a task that needs three retries to succeed costs three times what the happy path suggests, and if your pilot’s retry rate is high, that’s the actual finding, not a footnote. If you’re unsure what your unit cost even is, that’s the number to measure first, before any of the others. A consulting engagement or an audit through services usually starts exactly there, because it’s the number that kills budgets fastest and gets modelled last.
What actually separates the pilots that survive
The pattern across all four failure modes is the same: they are boring, foreseeable, and almost never fixed by a better model. Production readiness is process discipline — real data, per-step logging, a named owner, a cost ceiling — applied before the demo becomes a commitment, not after it becomes an incident. None of this is exciting to build. It is, however, the entire difference between a pilot that becomes infrastructure and one that quietly gets switched off six months later with nobody quite able to say why.