Most automation pitches survive exactly one question. They collapse the moment you ask a second. That’s not an accident, it’s a filter, and you should treat it as one. These are the automation vendor questions I ask on every call before a client signs anything, and the ones a competent seller answers without reaching for a slide.
I didn’t write these to be comprehensive. I wrote them to be hard to fake.
What breaks, and what happens when it does
Ask this first, because it’s the one most vendors haven’t thought through. Every automated system fails eventually — a model returns garbage, an API times out, a third-party rate limit kicks in — and what matters isn’t whether failure happens, it’s what the system does next. A vendor who says “it just works” hasn’t built error handling, they’ve built a happy-path demo. The honest version of this is a fixed order: retry, repair, escalate, and I’ve written about what each of those steps actually requires in more detail. If a seller can’t describe their version of that sequence off the top of their head, there isn’t one.
Is this actually an agent, or a pipeline wearing one
Ask directly, and don’t accept “it uses AI” as an answer. Most products sold as agents are fixed workflows with a model bolted onto one step, and that’s not automatically bad, but it changes what you’re buying and what it can adapt to. There’s a ten-minute test for this that you can run yourself before you commit to a contract, and it’s worth running even on a product you’re inclined to trust, because the label on the box rarely matches what’s inside it.
Is the underlying process actually fit to automate
This question gets skipped because it’s uncomfortable, and it’s the one that saves the most money. Automation doesn’t fix a broken process, it runs the broken process faster and at higher volume, and you find out it was broken once the errors are compounding weekly instead of surfacing once a quarter. Before you buy anything, map the process you’re paying to accelerate, and if nobody can draw it cleanly on a whiteboard, that’s the problem you need to solve first, not the vendor selection.
What does this cost per run, not per seat
License pricing hides the real number. The real number is cost per transaction once volume climbs, especially with anything calling a language model repeatedly, because token spend scales with usage in a way flat licensing fees don’t. Ask for a cost-per-run estimate at your actual volume, not the vendor’s example volume, and ask what happens to spend if a task runs long or loops. A system without hard caps on spend will eventually hand you a bill that makes the automation itself look like the cheap part.
Where does a human still have to sign off
This is the question that tells you whether anyone has thought about failure modes with real stakes attached. Not every step needs a person watching it, and plenty of “human in the loop” checkpoints in vendor demos are theater, there for compliance optics rather than actual risk reduction. Ask where the system stops and waits for a person, and ask what triggers that stop. If the answer is “never” and the task involves money, customer communication, or anything irreversible, that’s a design gap, not a feature.
What does a good proof of concept actually prove
Push for a proof of concept on your data, with your edge cases, not the vendor’s cleanest sample set. A convincing demo tells you the product can perform under ideal conditions. It tells you nothing about your actual lead volume, your messy CRM fields, or the client emails that don’t match any template. Ask what the vendor’s own acceptance criteria are for a successful pilot, and if they don’t have numeric criteria, write your own before the pilot starts, not after.
What happens after the contract is signed
Ask who owns the system once it’s live, who gets paged when it breaks at 2am, and how much of the configuration you can change without a support ticket. A lot of automation deals are sold by people who never touch the maintenance phase, and the gap between the sales conversation and the support reality is where most buyer’s remorse comes from. Get this in writing, specifically, before the demo ends.
The automation vendor questions that actually filter sellers
The twelve questions above work in both directions, which is the point. A vendor who answers them fluently, with specifics instead of reassurance, has run into these problems before and solved them. A vendor who gets defensive, vague, or pivots to a feature list hasn’t. These automation vendor questions test whether the person across the table has actually operated something like this under load, or just sold it.
Ask the uncomfortable one first. It saves everyone the rest of the meeting.