AI Hallucinations: Why Chatbots Make Things Up

You ask the tool for a study that backs up your idea. It hands you a precise title, the researchers' names, a year and an issue number. You paste the title into a search engine and find nothing. There is no such study and no such researchers. And all of it arrived in a calm, confident voice, as if the tool were reading from a book in front of it.

People call this "hallucination": a model presenting something with no basis in the form of a fact. It is not a rare glitch in weak models. Researchers from OpenAI and Georgia Tech wrote in a paper published in September 2025 that it persists even in state-of-the-art systems.

My own work includes reviewing AI output inside live systems, and what follows is what I ask the teams I work with to do: understand why the error happens, who has paid for it, where it tends to hide, and then follow a five-step check. This article is current as of October 2026, and every figure or event is sourced at the end.

Why does a model "hallucinate"? The simple version

In its basic form, a language model does not open a database and pull out the truth. It learns from a vast amount of text which words usually follow which words, then generates an answer that looks likely. When the question is about something it has seen many times, the likely answer matches the correct one. When the subject is rare, the answer stays likely in shape but may not be right.

The clearest recent explanation comes from Adam Kalai and colleagues at OpenAI and Georgia Tech, in a paper posted on September 4, 2025 called "Why Language Models Hallucinate." It explains hallucination from two angles: why it starts, and why it survives.

Why it starts: facts with no pattern

A person's birthday cannot be deduced from a rule or from logic. Either the model saw it often enough to store it, or it saw it once or never, and then it has no way to tell a true date from a made-up one that looks the same. The authors give an illustration: if 20% of birthday facts appear exactly once in the training data, you should expect the raw model to be wrong on at least 20% of them.

They tried it. They asked a large language model (DeepSeek-V3, in May 2025) for one of the authors' birthdays, and said to answer only if it knew. Across three attempts it gave three different dates, all wrong.

Why it survives: an exam that doesn't reward "I don't know"

Imagine an exam that gives a point for a correct answer and zero for a wrong answer and a blank page alike. The best strategy is to guess every time, because guessing costs nothing. The paper says most of the popular benchmarks used to measure models are graded this way. So a model that admits the limits of its knowledge loses to a similar model that guesses with confidence.

The authors called this an "epidemic" of penalising uncertainty and abstention, and proposed changing how the dominant tests are scored so that confident errors are penalised and saying "I don't know" costs nothing extra.

This does not mean the tool is trying to deceive you. It means it was trained and evaluated under conditions that reward the confident answer over admitting doubt, so its tone tells you nothing about whether it is right.

A test by the Tow Center for Digital Journalism at Columbia University, published in the Columbia Journalism Review on March 6, 2025, fits this picture. Researchers gave eight AI search tools verbatim excerpts from news articles and asked for each article's headline, publisher, date and link. Together the tools were wrong on more than 60% of queries. ChatGPT's search identified 134 articles incorrectly and signalled doubt only 15 times. Those are March 2025 results and the tools change, but the striking part is that the errors mostly came with no hint of hesitation.

Real stories, real prices

A lawyer asked the tool: is this case real?

In March 2023, two lawyers representing a man injured on a flight filed a brief in the Southern District of New York. It cited court decisions, and quotations from them, that did not exist, all produced by ChatGPT. When the airline's lawyers said they could not find the cases, the two lawyers did not back down.

Attached to a later affidavit, lawyer Steven Schwartz included screenshots showing that he had asked the tool itself whether Varghese was a real case. It said yes, and that it could be found on Westlaw, LexisNexis and the Federal Reporter. That is not verification. It is a question put to the thing that invented the case.

On June 22, 2023, Judge P. Kevin Castel issued his sanctions ruling. The punishment was not for merely using the tool: the judge wrote that there is nothing inherently improper about using a reliable AI tool, but that the rules make lawyers responsible for the accuracy of their filings. He found bad faith because the lawyers consciously avoided confirming the truth, made false and misleading statements to the court, and stood by the fake rulings after the court and the other side questioned them. He imposed a $5,000 penalty on the two lawyers and their firm jointly, and ordered them to write to their client and to the judges falsely named as authors of the fake opinions.

The judge also noted that the record would have looked quite different had the lawyers come clean when the other side first raised doubts. The first mistake was skipping verification. What led the court to find bad faith was doubling down.

A consulting report and a refunded instalment

Lawyers are not the only ones. According to Fortune's report of October 7, 2025, Deloitte Australia delivered a 237-page report to Australia's Department of Employment and Workplace Relations, published on the department's website in July 2025. A University of Sydney researcher flagged errors: references to academic papers that do not exist, a fabricated quotation attributed to a federal court judge, and a book that does not exist credited to a law professor.

A revised version dated September 26, 2025 then disclosed that a generative AI system, Azure OpenAI, had been used in preparing it. Fortune reports that the firm will repay its last payment instalment, which makes it a partial refund. Deloitte wrote that the changes did not affect the report's substance, findings or recommendations.

What the two stories share is that the tool signs nothing. Whoever signs is the one held to account.

These are not isolated incidents. Researcher Damien Charlotin tracks court decisions in which a court found, or implied, that a party relied on hallucinated material. As of October 10, 2026, his database held 2,184 cases from many countries, 864 of them involving lawyers. And the error does not always look like a phantom reference: the source may exist while it is credited with things it never says. The database tags cases by type of error: 1,791 with fabricated material, 922 with misrepresentation of real cases, and others with false quotations. The tags overlap, so one case can carry more than one.

Where hallucinations cluster: numbers, references, laws and names

Notice what these stories have in common: precise, checkable details such as a case number, a paper title, a word-for-word quotation. That fits what the Kalai paper says about facts with no pattern that rarely appeared in training data. These are the places I look first:

• Exact figures: a percentage, a price, a market size, a user count.
• References: a study title with its authors, year and journal.
• Laws and regulations: an article number, a decision number, a date of issue, the name of an authority.
• Names: lesser-known people or companies, and their biographies and job titles.
• Quotations and links: a sentence attributed to someone in quotation marks, or a link that doesn't open.

Researchers have measured this in law. When general-purpose chatbots were asked specific, verifiable questions about randomly chosen federal court cases, hallucination rates ranged from 58% with GPT-4 to 88% with Llama 2 (a study published in 2024). In another 2024 study, three commercial legal research tools that retrieve sources were wrong between 17% and 33% of the time. Those are numbers from that period, not today's rates, and the lesson is that connecting a tool to a database reduces errors without removing them.

Even when you put the text in front of the model and ask for a summary, the risk doesn't vanish. Vectara's leaderboard measures how often a model adds claims to a summary that the source doesn't support, across more than 7,700 articles. In its update of September 22, 2026, rates ran from 1.8% for the best model to 24.2% for the worst result, across more than 100 models. In the same table, a small model scored 3.1% while a "pro" model from the same developer scored 8.3%, so a higher tier is not automatically more accurate on this test. It measures summarisation only, and the leaderboard itself says determining hallucinations on any ad hoc question is impossible, because it isn't known precisely what data each model was trained on.

A five-step verification protocol

The rule I give teams: the tool drafts, a human verifies, and whoever signs is responsible. There are five steps, and they take minutes if you apply them only where they matter.

Not every document deserves all five steps

• Low risk (brainstorming, an internal draft): step one is enough, so you know which parts are unchecked.
• Medium risk (a deck for your boss, an email to a client): the first three steps for every figure and reference.
• High risk (a contract, a paid report, a legal document, a public post, a financial decision): all five, plus a second human reader.

From personal habit to team policy

If you manage people, the protocol alone is not enough. The team needs stated rules. These are the three I always set:

• Whoever signs a document owns what is in it, whichever tool wrote it.
• Important documents state that AI was used and what was checked by hand.
• Someone who says "I found a mistake" or "I haven't verified this yet" is thanked, not blamed.

The third rule matters most, and its lesson comes from the lawyers' story: owning up early is cheaper than doubling down. If an employee learns that saying "I haven't checked" brings blame, they will go quiet and polish the page instead. That kind of human behaviour inside teams is what I write about in The Blind Manager.

Reduce it from the start with how you ask

Back to the exam. If the tool behaves like a student afraid of leaving the page blank, write different exam rules into your question. These techniques come from Anthropic's guide to reducing hallucinations, and the company itself says they reduce hallucinations without eliminating them:

• Give the tool permission to say "I don't know." The guide describes this simple technique as one that can sharply cut false information.
• Attach the text and ask for quotations first. With long documents, have the tool pull out the relevant word-for-word sentences before analysing, then build only on them.
• Ask for support for every claim. Then ask it to retract any claim it can't back with a sentence.
• Limit it to what you gave it. Say explicitly: use only the attached documents, not general knowledge.

Here is a ready-made instruction to copy and adapt:

Answer only with what you are confident about. If you are not sure of a figure, name or reference, say "I don't know" and don't guess. Don't cite a quotation or source unless it appears in the attached text, and give the exact sentence you relied on.

Kalai and colleagues also suggested that people evaluating models state a confidence threshold and a penalty for errors in the question itself, such as "answer only if your confidence is above 75%." That is a proposal for testing models, not a guaranteed recipe for your chats, but it sums up the idea: change the exam rules. And don't assume that switching on web search relieves you of checking, because the news search tools in the Tow test and the commercial legal research tools were connected to sources.

The bottom line: what the tool writes is a draft, not a source

Hallucination is not a secret or a scandal. It is a result of how models are trained and evaluated, and Kalai and colleagues argue that the fix starts with changing how the dominant tests are scored. Until that happens, treat every figure, reference and quotation the tool writes as a draft awaiting your signature after verification, not as a source.

It resembles us more than we like to admit. Your own mind also fills a gap with a convincing explanation, in your own voice, and you believe it. In Istiqala min Nafsik ("Resigning from Yourself") there is a chapter on this tendency, "The Lawyer in Your Head," and its tool is close to what we have said here: write the facts before the explanation.