A language model does not know things. It produces text that fits, and most of the time text that fits is also text that is true — which is what makes the failures dangerous. When it is wrong, it is wrong in the same confident register it uses when it is right.
If that output is going in front of your customers, this is a business decision before it is a technical one, and it is one you can make without being technical yourself.
What "making things up" actually looks like
Rarely nonsense. Nonsense would be easy — you would spot it.
It looks like a plausible policy detail that is not your policy. A confident date. A refund figure that is close to right. A citation of a document that does not exist. The output is fluent, specific and well-formed, and it is the specificity that makes it convincing.
It also does not repeat. Ask the same question twice and you may get the correct answer both times, then the invented one on the third. This is why "we tested it and it was fine" is not evidence of much.
The four controls
You do not need to understand how they work to ask whether they are there.
Limits on what it is allowed to do. The model gets a narrow job and no access beyond it. It can answer from your published policies; it cannot issue a refund, change a record, or send an email. Most serious damage comes from an AI that was given the ability to act rather than merely to speak.
Grounding in your own material. Rather than answering from general knowledge, the system retrieves your actual documents and answers from those, with the source shown. This does not eliminate invention, but it makes it far rarer and — because the source is displayed — checkable.
Automated checking of the answers. A test suite, but for outputs: a set of real questions with known-good answers, run automatically whenever anything changes, with the results compared. Without this you cannot tell whether a change made things better or worse; you can only tell whether it still runs.
A person in the loop where it matters. Not everywhere — that would remove the point. Where the cost of being wrong is high: money moving, an entitlement decision, anything that would be expensive to retract.
The decision that actually matters
Before any of the above, answer this: what happens if it is wrong, and who bears it?
If a wrong answer means a mildly unhelpful reply and the customer asks again, ship it with grounding and checks. If a wrong answer means someone acts on incorrect information about their money, their entitlement or their obligations, the human stays in the loop, and no measured accuracy rate changes that.
The question is not "how often is it right?" It is "what does one wrong answer cost, times how often, against what it saves." That is arithmetic, and it is your arithmetic rather than your engineer's.
Five questions to ask before it ships
Show me it being wrong. Anyone who cannot produce a wrong answer on request has not looked hard enough.
What can it do besides talk? Every capability is a way for a mistake to become an action.
Where do its answers come from? "The model" is a worse answer than "these documents, cited".
How would we know if it got worse? If the answer describes a customer complaining, there is no monitoring.
What does the customer see when it fails? There should be a designed answer, not a blank screen or a fabrication.
If those questions are met with confidence rather than specifics, that gap is worth closing before launch rather than after. A Reality Check answers them in writing, in a week.
This is the part we do — the crossing from a demo to a system that survives production.