AI Chatbot Hallucination Audit: A Pre-Launch Checklist

The 60-second verdict

• Five failures block a launch: an unsupported capability claim, an unsourced number, a policy detail contradicting your current docs, an invented justification for an action already taken, and an irreversible action with no approval gate.
• Test-set size: 50-100 real questions pulled from your last 90 days of tickets, plus a smaller adversarial set covering things you do not offer and policies you changed this quarter.
• The gate: zero tolerance on the launch-blocking tier. One confirmed instance holds the launch. That sounds strict until you price the alternative. Air Canada paid $812.02 and a tribunal ruling over one wrong bereavement-fare answer.
• NIST's name for this is confabulation, one of twelve risk categories in its Generative AI Profile. NIST describes it as structural to how these models generate text. No vendor has fixed it, because it is a property of the technology they all build on.
• A vendor confidence threshold is not this audit. It catches cases where the model knows it is unsure. It does nothing for a fluently wrong policy answer, which is the Air Canada failure exactly.
• If an adversarial run of 110 questions finds nothing, your test set is too soft. That is the result most worth being suspicious of.

13.7% of verified G2 reviews in the AI Chatbots category mention accuracy or hallucination when asked what they dislike. That is the single most common complaint in the category, ahead of price, ahead of setup, ahead of support. The figure is measured across more than 2,950 reviews.

Most teams find this out after launch, from a support ticket, a screenshot on social media, or in one documented case a tribunal ruling. A pre-launch audit is cheaper than any of those, and it does not need to be complicated. It needs to check five specific things, in order of how much damage they do, before a real customer sees them.

This is that audit. Its categories come from NIST's own generative-AI risk taxonomy. It names two real incidents an audit like this would have caught.

What the complaint data actually says

G2's own analysis of its AI Chatbots category reviews states that inaccuracy and hallucination are the number one buyer complaint, ahead of cost. The figure behind that claim: verified reviews mention accuracy and hallucination problems 13.7% of the time when describing what they dislike, drawn from over 2,950 reviews in the category.

Read that number for what it is and not more. It is a share of dislike mentions. It is not an error rate. It does not say how often any given bot hallucinates, and it cannot be turned into a benchmark across vendors. G2 does not publish a per-vendor breakdown of that 13.7%, and no methodology here lets you back one out. What it does establish, reliably, is that this is the complaint buyers write down most often, which is a different and more durable fact than any single vendor's claimed accuracy rate.

The 13.7% figure is G2's own published analysis, cited as theirs.

What NIST calls this, and what its guidance actually does

NIST's AI Risk Management Framework is the reference document most vendor "responsible AI" pages gesture at without naming a section. Its generative-AI companion, AI 600-1, published in July 2024, is more specific. It gives the failure mode this page is about its own name: confabulation.

• NIST's definition: confabulation is generated content that's confidently stated but erroneous, false, or unsupported. The colloquial word for it is "hallucination." It is one of twelve generative-AI risk categories in the profile, alongside information integrity, harmful bias and data privacy.
• Why it happens: these models generate text by predicting the next token to match a statistical distribution. Nothing in that step checks a fact. NIST states plainly that confabulations "have been shown to be pervasive" in current state-of-the-art LLMs.
• What that rules out: the comfortable assumption that a good-enough vendor has solved this. NIST is describing a property of the technology every vendor here is built on. That is the argument for testing your own deployment. A marketing claim is not evidence about your own content.

What the RMF and the Generative AI Profile do not give you is a pass/fail number.

• NIST's own measurement pilot, ARIA, published its first evaluation report 13 November 2025.
• Covered three scenarios (TV Spoilers, Meal Planner, Pathfinder) across three evaluation levels: model testing, red-teaming, field testing.
• Five organisations submitted seven AI applications in total. That makes it a methodology paper. Nobody should read it as a benchmark.
• If the agency that wrote the risk taxonomy hasn't published a rate you can compare against, no vendor's own number should be taken as the industry standard either.

Two real cases an audit like this would have caught

Air Canada: how one wrong answer reached a tribunal

On 11 November 2022, Jake Moffatt asked Air Canada's website chatbot about its bereavement fare policy after a family death. The bot told him he could book a full-price ticket and claim the bereavement discount retroactively, within 90 days. Air Canada's actual policy required the discount to be requested before travel. When Moffatt applied afterward, he was refused and offered a $200 coupon instead.

The BC Civil Resolution Tribunal ruled against Air Canada on 14 February 2024, in Moffatt v. Air Canada, 2024 BCCRT 149. Air Canada's defence was that the chatbot was "a separate legal entity" responsible for its own words. The tribunal rejected that outright: it should be obvious to Air Canada that it is responsible for all the information on its website, and it makes no difference whether that information comes from a static page or a chatbot. Air Canada was ordered to pay $650.88 in damages, $36.14 in pre-judgment interest and $125 in tribunal fees. That comes to $812.02 in total, over a discount that would have cost it a fraction of that if the bot had simply repeated the real policy.

This is a decided case with a paid judgment behind it. The failure category is one an audit catches directly: a chatbot stating a policy detail, in this instance an eligibility window, that contradicts the vendor's own published terms.

Cursor: the bot invented the policy it was accused of violating

In April 2025, Cursor users on multiple machines started getting logged out unexpectedly, caused by a session-security bug. Users who emailed support got a confident answer from an AI agent signing its replies "Sam": logouts were expected behaviour under a new single-device policy. There was no such policy. Cursor co-founder Michael Truell confirmed it directly: "We have no such policy. You're of course free to use Cursor on multiple machines."

The invented explanation spread through a Hacker News thread and the Cursor subreddit faster than the correction did, and multiple users cancelled subscriptions over a rule that never existed. Cursor's fix afterward was procedural as well as technical: any AI response used in email support is now clearly labelled as such.

The Hacker News discussion, dated 15 April 2025, is worth reading past the headline. One commenter's summary of the failure is the one an audit is built to prevent: a support system that answers a factual question about company policy with an invented policy, stated with the same confidence as a true one, and no visible signal to the customer that a model wrote the answer.

A separate, smaller-scale version of the same failure shows up on OpenAI's own developer community forum, in a thread from 2 October 2023. A retrieval-augmented support bot began fabricating facts about a specific customer's account, asserting a school affiliation that was never in its knowledge base. The fabrication traced to conversation history biasing the model toward a plausible-sounding pattern instead of the retrieved facts. The practitioner reported that strict prompt instructions, even at temperature zero, did not stop it. What held was a structural fix: validating retrieved evidence and reformulating each turn as a standalone question. A better instruction was never going to be enough.

Three incidents, three different mechanisms: a wrong-policy answer, an invented policy stated as fact, and a retrieval failure disguised as a confident citation. An audit that only checks one of these will miss the other two.

What existing checklists already cover, and where they stop

Two published checklists get close to this already, and it's worth crediting what they do before adding to it. Linkup's is tight: seven checks, each testable in two to three minutes, covering source-claim traceability, freshness, entity resolution, numeric fidelity, calibrated abstention, cross-source contradictions and forced-format fabrication. Autonoma's is broader, at 34 items across five groups, and it carries a genuinely useful split. Functional and conversational checks run once before launch. The LLM-specific, security and performance groups belong in continuous integration on every deploy, because a model update can silently change behaviour that passed yesterday.

Neither checklist gives you a rule for when a bot fails badly enough that it should not ship. That is the gap this audit fills, and it is the gap both real incidents above sit in. Air Canada's bot and Cursor's bot would likely have passed a general functional QA pass. Neither would have passed a check built specifically to catch an invented policy stated as settled fact.

The audit: five categories, one severity rubric

Run every test query through your own bot and classify each failure by category, then by severity. The category tells you what broke. The severity tells you whether you can ship anyway.

The minimum-queries column adds to 110, close to the 100-query ceiling several vendor testing guides recommend as a practical pre-launch set.

Severity decides whether you ship. How bad an individual answer feels does not. "Blocks launch" means a failure you cannot explain away to a customer after the fact: a stated policy, a stated number, an action taken on invented grounds. "Fix before launch" means real but recoverable with a labelled disclaimer or a narrower scope. "Monitor post-launch" means track it, because it degrades slowly instead of failing loudly.

Building the test set without inventing the evidence

The audit is only as good as the questions in it, and the honest version of this section is short. Use real questions. Invented ones tell you nothing about your own content.

Pull your last 90 days of actual support tickets and sample from them directly: pricing questions, policy questions, edge cases like refund eligibility and plan changes. Vendor guidance on this converges on a workable floor. Assemble somewhere in the 50 to 100 range of real questions sourced from support tickets, sales transcripts or FAQ logs, weighted toward the categories your ticket volume already shows are common. That figure is a vendor's own recommendation.

Add a second, smaller set deliberately: questions about things you do not offer, policies you have changed in the last quarter, and numbers that exist nowhere in your documentation. This set tests abstention, where the main set tests accuracy. It is also the set most teams skip, because it is uncomfortable to write questions your own bot is supposed to fail.

One recommendation from vendor testing guidance is worth taking seriously. If your team cannot produce a hallucination by trying, your test set is not adversarial enough. A pre-launch audit that finds nothing wrong is much more likely to be too soft than genuinely clean.

The pass/fail gate: deciding what actually blocks launch

No industry-standard threshold exists for this. Nobody publishes one, including NIST, whose own measurement pilot covers seven applications total. So the gate has to be a decision you make and write down. A number borrowed from somewhere else will not survive contact with your own documentation.

A defensible starting rule: zero tolerance on "blocks launch" severity, a documented fix plan for every "fix before launch" item, and a named owner for every "monitor" item. Say your first run against a 110-question test set, sampled the way described above, comes back like the table below. It is a hypothetical worked example. It reports no measurement.

One blocking failure in a 110-question run is enough to hold the launch, under a zero-tolerance rule for that tier. That reads as strict until you compare it to the Air Canada case: one bereavement-fare answer, stated once, cost a company a tribunal judgment about 15 months later. The asymmetry between the cost of catching it in a 110-question test and the cost of a customer catching it in production is the entire argument for running the audit at all.

What vendors already build in, and what that does not cover

Some of this is partially handled upstream, and it is worth knowing where. Intercom's Fin AI Agent documents a genuine confidence mechanism. When handling email, Fin can pause on replies it is not confident about and ask a human teammate for input, instead of guessing or escalating immediately. A threshold controls it, set to Low, Moderate as the default, or High. That is a real, documented abstention mechanism, and it is closer to what NIST's confabulation guidance is asking for than most vendor pages get.

It is also narrower than it sounds. A confidence threshold governs the cases where the model itself is uncertain. It does nothing for the Cursor failure, where the model was confidently, fluently wrong, or the Air Canada failure, where a wrong policy detail was stated with full apparent authority. A vendor-side confidence gate is one layer. It is not a substitute for testing your own deployment against your own documents, because the vendor cannot know your policy changed last month unless your index says so.

What breaks in month three, after you buy

A pre-launch audit is a snapshot, and three of the five failure categories above degrade on their own after you pass it. This is the part teams skip, and it is why the abstention set matters more than the accuracy set.

• Policy drift is guaranteed. Your refund window, your plan names, your eligibility rules will all change. The stale-policy category is the only one in the audit that recurs on a schedule, and Air Canada's $812.02 judgment came from exactly this failure: a bot quoting a bereavement-fare rule that the airline's own policy contradicted.
• A model update can silently reverse a passing result. Autonoma's checklist makes this point, and it is the reason its LLM-specific group belongs in continuous integration, where a one-time launch gate would miss it. Behaviour that passed in March is not guaranteed to pass in June on a vendor-managed model you do not control.
• The knowledge base grows faster than the test set. The 50-100 questions were sampled from a 90-day window. Three months later, that window covers a different product, and nobody re-sampled.
• Citations rot quietly. The invented-citation category degrades into a broken-citation category: the URL the bot returns was real at launch and 404s by month three. It still looks like a confident, well-sourced answer.
• Re-run the abstention set first, ahead of the accuracy set. Ask again about the features you do not have and the policies you changed last quarter. A bot that has started answering those confidently has drifted in the one direction that produces a launch-blocking failure.

The verdict

Run this before launch. A customer finding the gap first is the expensive version. The audit above costs a team an afternoon against 50-100 real tickets. The two incidents in this piece cost one company a tribunal judgment and the other a wave of public cancellations, and both failures fall inside categories this checklist is built to catch.

Treat "blocks launch" severity as non-negotiable. A stale policy answer or an invented justification for an action is the exact failure mode NIST's Generative AI Profile names as structural to the technology. It is also the exact failure mode that has already produced a legal judgment against one company that shipped without catching it. Softening the wording in your next prompt revision addresses neither.

Do not mistake a vendor's confidence-threshold feature for having done this work. Fin's escalation threshold is real and worth using, but it catches only the cases where the model itself registers uncertainty. Stale documents and invented-but-fluent answers pass straight through it. That is precisely why the audit has to run against your own deployment and your own current documents. A vendor's marketing claim about its own model proves nothing about your content.

Sources read 17 September 2026. NIST's AI 600-1 profile and the ARIA pilot report are dated 2024 and 2025 respectively and have not been superseded by a later NIST publication as of this read; check nist.gov directly before citing either as current.

WRITTEN BY AR · UPDATED 2026-11-02

I read the fine print. Vendor pricing pages, billing definitions, terms, funding filings and acquisition notices — then I do the arithmetic nobody publishes: what a platform actually costs at your volume, what its headline metric is really counting, and who owns it now. I do not run benchmarks, and no page here pretends otherwise.

Editorial policy