Why AI Makes Things Up, and How to Catch It
Share
Hallucination is a bad name for it. The system is not seeing things. It is doing exactly what it was built to do.
Direct answer
AI invents things because it produces plausible text rather than looking anything up. A false citation is as plausible a continuation as a real one. It is a consequence of the design, so the fix is a checking habit rather than a better model.
Based on the AI and Automation Practitioner standard (ST1512), a Level 4 apprenticeship for people already in work. Funding rules described here are the England position and are set by the funding body.
Potentially free for you
The learner never pays. Employers potentially pay nothing either, and where the apprentice is aged 16 to 24 the government funds the whole cost even if the employer does not pay the levy.
Closing any gap. A levy transfer from another employer covers 100% of the cost. Unused transfer allowance is not carried forward at the end of the tax year, and it must be agreed before the apprentice starts.
From 1 August 2026 the government stopped adding the 10% top-up to new funds entering levy accounts, so there is less in the system than there was.
Part of our guide to AI Explained in Plain English for Work.
The mechanism
These systems produce what usually follows, given everything before it. Plausibility is the only criterion.
Truth and plausibility overlap most of the time, which is why the output is usually right and always confident.
Where they diverge, nothing in the system prefers truth. That is the entire explanation.
It fills gaps rather than flagging them, because a gap and a confident continuation look identical from inside the process.
Where it happens most
Citations and references. These have a strong recognisable form, which the system reproduces perfectly with invented content.
Numbers and statistics. Plausible-looking figures with no source, presented with the same certainty as real ones.
Names and attributions. Who said what, who wrote what, who works where.
Rules and entitlements. UK regulation, employment rights and funding rules are frequent failure points, often with American framing.
Where it happens least
Working from text you pasted in. Summarising, extracting and reformatting a supplied document is reliable.
Pure form tasks. Rewriting for tone, changing structure, matching a format.
Well-known general knowledge, where the pattern is so heavily reinforced that plausible and true have converged.
That distribution tells you exactly how to use it. Supply the facts and let it handle the form.
There is more on this in our guide to AI explained in plain English.
The thirty second check
Any number, any name, any date: verify against the source before it leaves your desk.
Any claim about rules or entitlements: find the official page. This is the highest risk category and the easiest to check.
Any citation: confirm it exists. Search the exact title. Fabricated references are common enough to assume rather than suspect.
Do it every time rather than when you feel unsure. Feeling unsure is not correlated with the output being wrong.
Why better models do not remove the need
Newer models invent less. The mechanism is unchanged, so the failure mode is rarer rather than absent.
Rarer is harder to catch. A system wrong one time in fifty produces complacency that a system wrong one time in five does not.
Retrieval features help genuinely, because they change the task from recall to reading. They do not eliminate it.
Plan for the habit to remain necessary indefinitely. Any other plan is a plan to publish something wrong eventually.
Where it gets UK facts wrong most often
These tools are trained on far more American text than British, and it shows in a specific and predictable way.
Employment rights, notice periods, holiday entitlement and redundancy rules come back with American framing surprisingly often.
So do tax and funding rules. Ask about apprenticeship funding and you may get a description of a US system with British words attached.
Treat anything about UK regulation, entitlements or funding as unverified until you have found it on a gov.uk page or a regulator's own site.
What to do when it happens
Do not argue with it. Pointing out an error often produces a confident correction that is also wrong.
Start again with the source material supplied. This changes the task rather than the instruction.
Record the error if it reached a document. Knowing where a wrong figure entered matters more than fixing that one instance.
If a wrong output reached a customer, treat it as an incident with a cause rather than a one-off.
Key takeaways
- It produces plausible text, and plausibility is the only criterion.
- Citations, numbers, names and UK regulation are the highest risk categories.
- Working from text you pasted in is reliable. Recall is not.
- Check every time, because feeling unsure does not correlate with being wrong.
- Do not argue with it. Start again with the source supplied.
Frequently asked questions
Why does AI make things up?
It produces plausible text rather than looking anything up. A false statement is as plausible a continuation as a true one, and nothing prefers the truth.
What is an AI hallucination?
The industry term for confident invention. It is a consequence of how these systems work rather than a fault being fixed.
Where is AI most likely to be wrong?
Citations, numbers, names and attributions, and anything about UK rules or entitlements, where American framing frequently appears.
Where is it reliable?
Working from text you supplied, pure form tasks like rewriting for tone, and well-established general knowledge.
How do I catch invented facts?
Verify any number, name, date or citation against the source. For rules and entitlements, find the official page rather than trusting the answer.
Do newer models still make things up?
Yes, less often. Rarer is harder to catch, because it produces complacency that frequent errors do not.
Should I tell it that it was wrong?
Not usually. Pointing out an error often produces a confident correction that is also wrong. Start again with the source supplied.
Does retrieval fix the problem?
It helps genuinely, because it changes the task from recall to reading. It reduces the failure rate rather than removing it.
The funded route, in figures
Figures are the England position. Levy-paying employers usually draw the whole cost from the levy they already contribute, so they pay nothing extra. Employers below the £3 million pay bill threshold pay nothing for an apprentice aged 16 to 24. For an older apprentice the government funds 95 per cent, and a levy transfer from a larger employer can cover the rest. Eligibility and funding rules are set by the funding body and can change.
Where to go next
If you want checking habits and responsible use assessed as part of a qualification, the funded route covers both.
We help UK employers and employees access government-funded AI training. Training is delivered by third-party providers. We do not guarantee a place, a qualification, a job or any particular outcome. Eligibility and funding rules are set by the funding body and can change. Always check the current position before you commit. See our disclaimer.