A diligence list for leaders asked to fund an AI pilot. Most failed pilots fail for reasons that were visible in the first conversation.
Pilots are cheap to start and expensive to abandon. The direct cost is modest; the real cost is the credibility spent — after two inconclusive pilots, the next proposal gets a harder hearing even when it is a good one.
Most of the pilots that go nowhere were diagnosable at the outset. These are the questions I ask, roughly in the order I ask them.
1. What decision or task are we changing, specifically?
Not “improve customer service.” Which task, performed by whom, how many times a day?
If the answer stays at the level of a capability — “we want to use AI for support” — the pilot has no definition of done, and it will end when attention moves elsewhere rather than when a question is answered.
2. How many times does this happen per week?
Volume is the single best predictor of whether an AI use case is worth the effort. A task performed 4,000 times a week tolerates a lot of implementation cost. One performed eleven times a week almost never does, regardless of how annoying it is.
Ask for the number. If nobody has it, that is itself a finding — you are about to automate a process nobody measures.
3. What happens when it is wrong?
Every AI system has an error rate. The question is who absorbs it.
Wrong meeting summary: someone reads the transcript. Wrong ticket categorization: a reassignment, mildly annoying. Wrong answer about a patient’s medication, a customer’s contract terms, or a candidate’s eligibility: a materially different situation requiring different controls, and possibly a reason not to proceed.
If the proposal does not have an answer, it has not been thought through.
4. Does the data exist today, in a usable state?
Not “could we get the data.” Does it exist now, is it accessible to the team doing the work, and is it consistent enough to be useful?
This is where most pilots quietly fail. They are scoped assuming data that turns out to be spread across three systems that disagree, or locked behind an access review that takes a quarter. The pilot then spends its budget on data plumbing and produces no verdict on the actual question.
5. Who owns the process we are changing, and are they in the room?
If the owner of the affected process has not been part of the conversation, the pilot will produce a technically working result that nobody adopts.
This is not a formality. The process owner is the person who has to tell their team to work differently, defend the change when it goes wrong in week two, and give up a workaround they have relied on for years. If they are not invested, you are funding a demo.
6. What is the baseline, and have we captured it?
Whatever you intend to improve — handle time, deflection rate, cycle time, error rate — you need the current number before the pilot starts.
This gets skipped constantly, and it is unrecoverable. Without a baseline, your post-pilot conversation becomes an argument about whether things feel better, which the pilot will lose against anyone who was skeptical to begin with.
7. What result would make us stop?
Define this in advance and write it down.
A pilot without a stop condition does not conclude; it just persists. Agreeing beforehand that “below X accuracy, or below Y adoption after Z weeks, we stop” turns a political argument into a pre-agreed decision. It also makes the pilot honest — if you cannot name a result that would make you stop, you are not testing anything.
8. What does it cost to run at full scale, not pilot scale?
Pilot economics mislead in both directions. Consumption-priced AI looks cheap at 200 calls a day and is a different conversation at 60,000. Conversely, build effort amortizes well at scale.
Model the run cost at realistic volume: licensing, consumption, infrastructure, and the human review time the design requires. If the full-scale number is unaffordable, the pilot’s only possible outcome is a disappointment.
9. Who will own this in production?
A pilot is owned by whoever is excited about it. Production needs someone whose job includes it — monitoring it, handling its failures, updating it when the underlying content or process changes.
If there is no named production owner, you are building something that will work for six months and then slowly rot, which is worse than not building it, because it will be quietly trusted while it degrades.
What a good answer set looks like
A proposal that clears this list sounds like: we are changing how first-line agents triage password-reset requests, which happens around 900 times a week; a wrong categorization costs a reassignment and is caught the same day; the ticket history exists in our platform and we have five years of it; the service desk manager is sponsoring it and will change the work instruction; current average triage time is 4.2 minutes and we have pulled the data; if we cannot reach 85% agreement with human categorization on a held-back sample, we stop; at full volume the run cost is roughly $X per month; the service desk manager owns it in production.
That is not an exciting proposal. It is a fundable one.
Any figures in this post are illustrative examples, not results we have delivered for a client.