SaaS Compare
A procurement checklist for AI tools
Twelve questions that separate vendors who have run this in production from vendors who have run it in a demo. Steal them for your next evaluation call.
- Author 01
WebTechOS Editorial
- Published 02
2026-06-22
- Read 03
6 min
AI vendor evaluation has a specific pathology: the demo is always excellent, because the demo was built on data the vendor chose. The questions below are designed to find the edges quickly, and most of them are uncomfortable enough that the answer's quality matters less than the vendor's willingness to answer at all.
On capability
Show me a failure case from a real customer deployment, and what you changed afterwards. Vendors without a good answer here have not run at scale.
What does the system do when it does not know? Confident fabrication versus explicit abstention is an architectural choice, and you want to know which one you are buying.
How do I reproduce a specific bad output from three weeks ago? If the answer involves grep and hope, plan to buy observability separately.
On commercials
What exactly counts as a billable unit, in writing? Every disputed invoice in this category is a definitional dispute.
What is the bill if volume triples in one month? Ask for the number, not the reassurance.
What happens at renewal if usage fell? Outcome-priced contracts often have floors that make a successful efficiency programme expensive.
Ask for a real failure case. The vendors who cannot produce one have not run this at scale.
On risk
Where does my data go, how long is it retained, and is any of it used for training? Get it in the contract, not the trust centre page.
Can I export everything, including the configuration and the history, in a usable format? Test the export during the trial rather than during the divorce.
Which model provider is behind this, and what happens if they deprecate it? Model deprecation is now a routine operational event and vendors should have a documented answer.
What is the blast radius of a wrong answer, and what controls limit it? This is the question that should determine how much human review you keep, and it is the one most often skipped.
WebTechOS accepts no payment for coverage, placement or scores. Where a piece references pricing, it reflects published list rates at the date shown.
Keep reading
All insights →The agent pricing reckoning
Per-seat licensing assumed software helps a person work. Agents do the work. Four billing models are now fighting to replace the seat, and three of them will lose.
AI Agent HubWhy most agent projects fail in month three
The demo works. The pilot works. Then something changes upstream, nobody notices for two weeks, and trust never comes back. A field guide to the failure that kills more agent projects than capability ever has.
SaaS InsiderThe consolidation nobody predicted
Unbundling was supposed to be the permanent condition of SaaS. Instead, buyers spent two years cutting vendor counts, and the tools that bundled well are winning categories they were not supposed to.