AI Hallucination: Real Cases Where Confident AI Answers Cost Money

By the Cehpoint teamPublished 6 min read

Short answer: AI hallucinations are confident but false outputs, such as invented court cases or refund policies. Documented cases include fake citations in Mata v. Avianca, Air Canada being held liable for its chatbot's wrong advice, and a Bombay High Court rebuke over AI-generated judgments in a tax order. The risk drops when answers are grounded in your own documents, citations are verified, and humans review high-stakes output.

Large language models are very good at producing text that sounds right. That is different from producing text that is right. When a model fills a gap with something plausible but false, such as a court case that never existed, a refund policy the company never had, or a figure nobody published, we call it a hallucination.

Most hallucinations are harmless. Some end up in a court filing, a tax order or a customer conversation, and then they cost money and reputation. Below are three documented cases, what actually went wrong in each, and the engineering and process patterns that reduce the risk in your own business.

Case 1: Fake legal citations in Mata v. Avianca

In a personal-injury case against the airline Avianca in the US District Court for the Southern District of New York, the plaintiff's lawyers filed a brief that cited court decisions generated with ChatGPT. As summarised on Wikipedia's page on the case, the filing contained fictitious cases, invented quotations and fabricated internal citations. Judge P. Kevin Castel sanctioned the lawyers, who were ordered to pay a $5,000 fine.

Root cause: the tool was used as if it were a legal database. It is not. A general chatbot generates text that matches the pattern of a citation; it does not look one up. Nobody in the chain opened the cited judgments before filing.

Case 2: Air Canada's chatbot invents a refund policy

In November 2022, Jake Moffatt asked Air Canada's website chatbot about bereavement fares after his grandmother died. According to CBS News, the chatbot told him he could apply for a partial refund if he applied within 90 days. The airline's actual policy required bereavement fares to be requested before travel. When he applied, Air Canada refused.

At British Columbia's Civil Resolution Tribunal, Air Canada argued that the chatbot was "a separate legal entity that is responsible for its own actions". The tribunal rejected that, holding that the chatbot is part of the airline's website and that it makes no difference whether information comes from a static page or a chatbot. Air Canada was ordered to pay about CA$812 in damages and fees.

Root cause: the bot answered a policy question from general knowledge instead of from the company's own policy text, and nothing checked that its answer matched the official policy. The money was small; the precedent was not. A business is responsible for what its bot tells customers.

Case 3: An Indian tax order built on judgments that do not exist

India has its own example. As reported by Bar & Bench, in KMG Wires Private Limited v. AO, NFAC, the Bombay High Court rebuked an assessing officer of the National Faceless Assessment Centre who had cited non-existent, AI-generated judicial decisions while raising a ₹27.91 crore tax demand. The court stressed that quasi-judicial authorities cannot blindly rely on AI-assisted searches or automated results when deciding a taxpayer's rights.

Root cause: the same as Mata v. Avianca, from the other side of the table. Output was trusted because it looked authoritative.

Why do AI models hallucinate?

A language model predicts the most likely next words given everything before them. It has no built-in notion of "I don't know". When the right answer is not strongly represented in what it learned, or not provided in the prompt, it still produces fluent text, and fluent text is persuasive. Hallucinations become more likely when:

  • The question asks for specifics: citations, numbers, dates, names, policy clauses, part numbers.
  • The answer depends on private or recent information the model never saw, such as your refund policy or last month's price list.
  • The prompt pressures the model to always answer, with no permitted "I can't find that".
  • Nobody checks the output before it reaches a customer, a court or a regulator.

OWASP lists this risk as LLM09, Misinformation, in its Top 10 for LLM applications. You cannot train it out entirely. You can design around it.

Patterns that reduce hallucination risk

1. Ground answers in your own documents

Retrieval-augmented generation (RAG) fetches relevant passages from your approved sources, such as policy documents, product manuals and price lists, and instructs the model to answer only from them. Done properly, the Air Canada answer would have come from the actual bereavement policy page. Grounding is only as good as the documents: keep one current version of each policy and retire old ones.

2. Require citations and check them automatically

Make the system return the source passage or document ID for every factual claim, and have code verify that the cited passage exists and actually contains the claim. If a citation cannot be verified, the answer is not shown. For legal or tax work, the system should link to the judgment in a real database, and a person must open it.

3. Allow, and reward, "I don't know"

Instruct the model to say when the sources do not contain the answer and to hand over to a human. Measure it. A support bot that honestly hands over the questions it cannot answer is far safer than one that answers every question confidently.

4. Match human review to the stakes

UseRisk of a wrong answerRecommended control
Internal brainstorming, first draftsLowUser judgement; label as AI draft
Customer support on general topicsMediumGrounded answers, citations, easy handover to a human
Prices, refunds, policy commitmentsHighAnswer only from approved text; templated responses for commitments
Legal, tax, medical, financial advice or filingsVery highAI assists research only; a qualified person verifies every source and signs off

5. Monitor in production

Hallucinations show up over time, not in a demo. Log questions, retrieved sources and answers. Sample conversations every week for accuracy. Track answers that cite no source, customers who come back with the same question, and topics where the bot frequently says "I don't know", since those reveal gaps in your documents. Re-test whenever the underlying model, prompt or document set changes.

6. Write a clear AI-use rule for staff

Many hallucination incidents are not chatbot failures but people using a general AI tool for work it was never designed for. A one-page rule helps: AI output is a draft; every fact, number, quotation and citation must be verified against an original source before it goes to a client, court, regulator or the public.

A note for Indian businesses

MeitY's March 2024 advisory to intermediaries asked that under-tested or unreliable AI models be offered to Indian users only with a label about their possible fallibility. Whatever the legal status of that advisory for your business, the principle is sound: tell users when they are talking to AI, and do not let it make commitments a person would not make.

How Cehpoint can help

Our LLMOps and AI observability service puts the monitoring described above into production: logging of prompts and sources, hallucination checks, quality sampling and prompt versioning, so you see problems before your customers do. If you are building a new assistant, our AI development team designs grounded, citation-checked systems from the start, and our AI red teaming service tests how easily it can be pushed into confident wrong answers.

Frequently asked questions

Is a company legally responsible for what its AI chatbot says?

In the Air Canada case, the British Columbia Civil Resolution Tribunal held that it was. The airline argued its chatbot was a separate entity, and the tribunal rejected that, treating the chatbot as part of the company's website. Courts elsewhere may reason differently, but the safe assumption for any business is that customers will hold you to what your bot tells them.

Does RAG stop hallucinations completely?

No. Retrieval-augmented generation greatly reduces hallucination by giving the model your actual documents to answer from, but the model can still misread a passage, combine sources wrongly or answer when the right document was not retrieved. Pair RAG with citation checks, permission to say I don't know, and regular sampling of real conversations.

Can we use ChatGPT or similar tools for legal or tax research?

They can help with first drafts and ideas, but never treat their citations as real until you have opened the judgment or provision in an authoritative database. The lawyers in Mata v. Avianca were sanctioned for filing AI-invented cases, and the Bombay High Court criticised a tax officer for relying on AI-generated judgments that did not exist.

How do we measure hallucination in our chatbot?

Build a test set of real questions with known correct answers and sources, run it after every change, and score accuracy and citation validity. In production, log answers with their retrieved sources, sample conversations weekly, and track unsourced answers and repeat questions. These numbers show whether the system is improving or drifting.

Need help with this?

Tell us what you are dealing with. We reply with a plain plan, and where it makes sense, a working demo before you commit.