How to measure an AI pilot before expanding the budget

Analysis and checklist. An AI pilot should answer a purchasing question: is this system useful enough, at its actual operating cost, to keep running? The following is a proposed evaluation method, not a report of measured customer results.

Choose the unit of success

Use a completed business task as the unit: a correctly routed enquiry, an approved document or a resolved support case. Count work that still requires correction. A system that generates ten drafts has not necessarily completed ten tasks.

Keep a comparison set from the existing process. Include ordinary cases, ambiguous cases and cases that should be rejected. Avoid testing only examples selected because the demo handled them well.

Keep four separate measures

• Acceptance: was the result correct enough to use?
• Effort: how much human correction remained?
• Cost: include inference, retrieval, retries and review.
• Failure: what happened when the system could not finish?

Set the decision rule before reading the results. For example, require the new process to match the existing acceptance standard while reducing total handling effort. The numerical threshold should come from your operation, not a generic industry promise.

Assign responsibility for failures

The NIST AI RMF Playbook offers voluntary actions organised around Govern, Map, Measure and Manage. It is a reference for structuring risk work, not a certificate that a pilot is safe.

For a small deployment, our recommendation is to name an owner, document the allowed actions and preserve a rollback path. If performance deteriorates after a model change, the team should know who can pause the system and how unfinished work returns to the normal process.

Architecture choices also matter: see agent versus workflow.

Put this into practice with the free AI evaluation worksheet.