Every vendor says they use AI, so the question "do you use AI?" now sorts nobody from anybody. These ten do. They are written to be handed to a supplier as-is, and the useful signal is often the shape of the answer rather than its content.
The ten
1. Show me what happens when it's wrong. Everything follows from this. A team that has run in production has a specific answer for a specific failure. A team that has not will describe how careful they are.
2. Where does our data go, and who else sees it? Name the providers. A vendor who cannot immediately list their subprocessors has not been asked before, which tells you about their other customers.
3. Is our data used to train anything? Should be an unambiguous no, in the contract, including for their upstream providers.
4. How do you keep our data separate from other customers'? Listen for whether isolation is enforced in the data layer or only in the application. "Every query filters by customer" is an application answer, and it fails the day one query forgets.
5. What can the AI do besides produce text? Any ability to act — send, pay, change, delete — turns a wrong answer into a wrong action.
6. How do you know if quality drops? The strong answer describes an evaluation set run automatically on every change, with alerting. The weak one describes customers telling them.
7. What record is kept of each interaction, and for how long? If you are regulated, you need inputs, outputs, versions and timestamps to be retrievable. Ask to see one.
8. Which model, and what happens when it changes? Models are deprecated and updated on the provider's schedule. A vendor without a plan for that has a dependency they have not thought about.
9. What's your security evidence? SOC 2, ISO 27001, a recent pentest — or an honest "we're pre-audit, here's what we've built." The honest answer is fine. A vague one is not.
10. If we leave, what do we take? Export format, notice period, deletion timeline and confirmation. Ask at purchase, when you have leverage.
How to read the answers
Specifics beat reassurance. "We take security very seriously" is not an answer to any of the ten. A named control, a named provider, a named document is.
Documents beat intentions. A team that has crossed into production has artefacts — a policy, an architecture note, a test report. A team that has not will describe plans in the present tense.
Speed matters. These should be quick. A vendor who needs a week to answer question 2 does not have a data map.
Honest limits are a good sign. "We don't do that yet" is more reassuring than a yes to everything. Nobody has all ten perfect, and a supplier who claims otherwise is either not listening or not telling the truth.
Give them to your compliance officer
These map onto what a compliance function needs anyway: data flow, subprocessors, retention, isolation, evidence, exit. Handing them over converts a technical evaluation into one your second line can sign off, which is usually the actual bottleneck.
And apply them to us as readily as to anyone else. We would rather answer ten questions properly than win work that unravels at diligence. For firms where getting this wrong is expensive, the answers are the product — and a Reality Check is a week spent producing them about whatever you already have.
This is the part we do — the crossing from a demo to a system that survives production.