The most expensive AI pilot is the one that proves the model can produce an impressive answer without proving that the workflow is better, safer or cheaper. Start with the decision the pilot must support, then design the smallest test that can change it.
Define the decision first
Write one sentence: “After this pilot we will decide whether to continue, expand or stop based on this evidence.” Name the owner of that decision. If no decision depends on the result, the project is a demonstration, not a pilot.
- The user and workflow that should improve.
- The decision the evidence will inform.
- The cost, latency and risk boundaries that matter.
- The person accountable for continuing or stopping.
Choose a bounded workflow
- Inputs are available and permitted for this use.
- A human can review the output without redoing the whole task.
- Failure is recoverable and does not silently change a critical record.
- The current process has a baseline you can compare against.
- The workflow does not require every edge case on day one.
Data and permissions before prompts
Decide which data may enter the system, where it is stored, how long it is retained and who can see it. Do not paste production personal data into a tool because the pilot is “only a test”. A pilot that creates a privacy incident is not a successful pilot.
- Data classification and the minimum data needed for the task.
- Retention, deletion and access rules for prompts, outputs and logs.
- Whether the provider can use the data for training and how to disable it where required.
- A path for a person to correct or delete a record.
- A clear owner for security and privacy questions.
Evaluate before you polish
- Build a small evaluation set with representative and difficult cases.
- Define what “good” means before looking at model output.
- Classify failures: missing context, wrong rule, unsafe action, formatting or refusal.
- Compare against the current process, not only against another model.
- Measure human review time, latency and cost per completed task.
- Test the guardrails: permissions, confirmation steps and recovery from mistakes.
Security and human control
- Least-privilege access to tools, files and APIs.
- Explicit confirmation before sending, deleting, paying or changing a record.
- Logs that show the input, the action and the reviewer without exposing unnecessary personal data.
- A fallback when the model is unavailable or uncertain.
Pilot checklist
| Area | Question to answer |
|---|---|
| Use case | Which workflow improves, for whom, and how will we notice? |
| Baseline | What does the current process cost in time, errors and review effort? |
| Data | What is the minimum permitted data, and where does it go? |
| Evaluation | Which cases represent success, ambiguity and failure? |
| Human review | Who checks the output, and what can they correct? |
| Risk | What happens when the model is wrong, unavailable or manipulated? |
| Economics | What is the cost per completed task, including review? |
| Decision | What evidence leads to continue, change or stop? |
From pilot to production
Production adds real users, changing data, monitoring and support. Before scaling, confirm that the evaluation set still represents production, that someone owns failures and that the system can be turned off safely. A pilot is evidence; production is an operating commitment.
See AI integration, AI integration risks or discuss your project.