1. The task is defined, and so is “correct”
“Use AI for purchase orders” is not a task. “Read incoming purchase orders and fill in nine fields in the ERP: customer, delivery address, delivery date, payment terms and, for each line, SKU, quantity and price” is. So is the definition of success: the record matches what an experienced colleague would have entered.
Without that definition there is nothing to measure, and a workflow that cannot be measured cannot be trusted.
2. It has been measured on your own documents
A model that performs well on clean examples will meet scanned faxes, handwritten notes, multi-page orders and the customer who always sends a spreadsheet. Build an evaluation set from real examples, including the awkward ones, with the correct answer recorded for each. A few hundred documents is usually enough to see the pattern.
Run the workflow against that set and score it field by field. Then run it again on every change to the prompt, the model or the rules. Quality becomes a number you track, not an impression from the last demo.
3. You know how it fails, not just how often
An average accuracy hides what matters. Ask two questions in plain terms:
- Of the orders it marks as ready, how many are actually right? This tells you how much you can rely on its confidence.
- Of the orders that are fine, how many does it let through without help? This tells you how much work it really saves.
A worked example, with illustrative numbers. Out of 300 test orders, the workflow marks 240 as ready, and 236 of those are correct: when it says ready, it is right 98% of the time. Of the 270 orders that were actually straightforward, it handled 236 without help: it saves the work on 87% of them. The remaining orders go to a person, which is the point.
Look at which fields fail and why. A workflow that is always right about quantities and sometimes wrong about delivery dates needs a different safeguard from one that fails at random.
4. The approval boundary is agreed
Decide, with the people responsible for the process, what the system may do on its own and what it may only prepare. At the start the answer is usually: it prepares everything and commits nothing. A person reviews each draft, and anything below a confidence threshold, or outside the rules, goes to a person first.
The boundary can widen later, one category at a time, when the measurements justify it. It should never widen because the pilot felt good.
5. It is monitored, and there is a way back
In production, log every input, output and approval. Track how often people override the system and why; that rate is the most honest measure of quality you will have. Re-run the evaluation set whenever anything changes.
And keep a way back. Any workflow should be switchable to manual handling at any moment, without losing the work in progress.
A rollout that respects all five
- Shadow mode. The workflow runs alongside the team, who keep working as before. Its output is compared with theirs.
- Approval mode. The workflow prepares every case and a person approves each one. Overrides are logged and reviewed.
- Selective automation. Only categories with a long, stable record of low overrides move to lighter review.
Questions to ask whoever builds it
- What exactly is the task, and how do we know a result is correct?
- How many of our real documents was it tested on, and can we see the results by field?
- What happens when it is unsure, and who sees those cases?
- What is it allowed to do without a person’s approval?
- How will we know if quality drops next month, and how do we switch it off?
Clear answers to all five are a good sign. Vague answers usually mean the workflow has only been tested on the examples in the demo.