A procurement checklist for AI tools

On capability

Show me a failure case from a real customer deployment, and what you changed afterwards. Vendors without a good answer here have not run at scale.

What does the system do when it does not know? Confident fabrication versus explicit abstention is an architectural choice, and you want to know which one you are buying.

How do I reproduce a specific bad output from three weeks ago? If the answer involves grep and hope, plan to buy observability separately.

On commercials

What exactly counts as a billable unit, in writing? Every disputed invoice in this category is a definitional dispute.

What is the bill if volume triples in one month? Ask for the number, not the reassurance.

What happens at renewal if usage fell? Outcome-priced contracts often have floors that make a successful efficiency programme expensive.

Ask for a real failure case. The vendors who cannot produce one have not run this at scale.

On risk

Where does my data go, how long is it retained, and is any of it used for training? Get it in the contract, not the trust centre page.

Can I export everything, including the configuration and the history, in a usable format? Test the export during the trial rather than during the divorce.

Which model provider is behind this, and what happens if they deprecate it? Model deprecation is now a routine operational event and vendors should have a documented answer.

What is the blast radius of a wrong answer, and what controls limit it? This is the question that should determine how much human review you keep, and it is the one most often skipped.