AI Hallucinations: When Your Assistant Confidently Makes Things Up
Otieno, a young advocate in a busy Nairobi practice, was rushing a memo late one evening. He asked an AI tool to find a supporting authority on a point of contract law. It gave him a case name, a citation and a neat summary of the judgment. Beautiful. The only problem? The case did not exist. Neither did the citation. The AI had invented the whole thing, and it had done it with the calm confidence of a senior partner. This is what people mean by AI hallucinations in business, and it is one of the most important things to understand before you trust these tools with real work.
What exactly is a hallucination?
AI language tools work by predicting what text is likely to come next, based on patterns learned from huge amounts of writing. They do not look up facts the way a librarian would. Most of the time, the patterns produce accurate answers. But when the tool does not know something, it rarely says "I do not know". Instead, it produces something that sounds right: a plausible name, a believable number, a realistic quote.
That is the dangerous part. A hallucination does not look like a mistake. It looks exactly like a correct answer.
Where AI hallucinations in business cause the most damage
Some uses are low risk. If AI suggests a slightly odd name for your new product, nobody is hurt. Other areas need far more care:
Legal research: invented cases, sections or regulations
Tax and compliance: outdated or made-up rules, rates and deadlines
Financial figures: wrong calculations presented neatly in a table
Health and safety information, where errors can hurt people
Quotes and statistics in reports, marketing or press releases
Details about real people or companies, which can become a defamation problem
If you would be embarrassed, fined or sued for getting it wrong, do not rely on AI alone.
Why it happens more than you might expect
Hallucinations become more likely when you ask about very specific, local or recent information. Kenyan laws, county regulations, recent KRA changes, local court decisions and small Kenyan companies are all areas where AI tools may have little reliable information. They will still give you an answer. It just may be fiction.
The way you ask matters too. Leading questions like "Give me three cases that support my argument" practically invite the tool to make something up, because it wants to give you what you asked for.
How to protect yourself
You do not need to stop using AI. You need a few habits:
Verify every fact, figure, name and citation against an original source
Ask the tool for its sources, then check that those sources actually exist and say what it claims
Use AI to explain, draft and structure, not as your final source of truth
Give it your own trusted documents to work from, rather than relying on its memory
Be extra careful with anything local, recent or highly specific
Make a rule that AI-assisted work gets a human review before it leaves the office
Building checks into your systems
For businesses using AI inside their own software, good design reduces hallucinations. Tools can be set up to answer only from your approved documents, to show where each answer came from, and to say clearly when they cannot find something. That is far safer than a general chatbot guessing. If you are thinking about adding AI to your business systems in a responsible way, our custom software development services cover exactly this kind of careful build.
Back to Otieno
Otieno caught the fake case only because his senior, Nyambura, asked for a copy of the judgment. He searched for twenty minutes before realising it did not exist. These days, he still uses AI to help structure his memos and summarise long documents, but every authority gets checked in the proper law reports. "It is like a very clever friend who sometimes lies to impress you," he says. "Useful, but you verify everything."
Your next step: next time an AI tool gives you a fact you plan to use, spend two minutes confirming it from the original source. Those two minutes are the cheapest insurance you will ever buy.