- The "AI Made It Up" Defence Is Running Out of Road
- The Real Failure Happens After the Hallucination
- Better AI Has Not Eliminated the Problem
- Courts Are Starting to Build Verification Into Filing Rules
- The Mandatory Citation Check
- The Human-in-the-Loop Is Not a Slogan
- The Bigger Governance Problem
- The Risk Is Not Just a Bad Citation
- The Lesson for Law Firm Leaders
- AI Policy Should Become AI Procedure
- The Most Dangerous AI Error Is the One That Gets Filed
- Conclusion
Two recent cases show how little room courts are leaving for lawyers who blame AI after filing work containing fabricated authorities.In the United States, the Eleventh Circuit referred attorney Anthony Sabatini for potential discipline after finding that his appellate briefs contained multiple nonexistent and hallucinated citations. The court described the filings as containing fake authorities and referred the matter to its Committee on Lawyer Qualifications and Conduct.
In the United Kingdom, the Crown Prosecution Service (CPS) admitted that AI-generated hallucinated cases had been included in written submissions to the High Court in an extradition appeal. The court accepted the CPS’s apology but made the underlying point very clearly: the immediate source may have been generative AI, but the operative cause was human failure to verify the authorities. The errors were discovered before the hearing and did not affect the court’s judgment.
These cases are important for a reason that goes beyond AI.The technology may have changed. The lawyer’s responsibility has not.
The “AI Made It Up” Defence Is Running Out of Road
This problem first became highly visible in 2023, when lawyers submitted briefs containing fabricated case law generated by ChatGPT.Three years later, the same fundamental failure is still happening.But the environment has changed.AI systems are now being used far more extensively across legal practice.
The output is more convincing.The workflows are more sophisticated.And the cases in which errors are appearing increasingly involve serious litigation and public institutions.
The uncomfortable question is therefore no longer:
“Can AI hallucinate?”
We already know the answer.
The question is:“Why did the lawyer allow an unverified AI output to become a court filing?”
The Real Failure Happens After the Hallucination
An AI system generating a fictional case is a technical failure.A lawyer filing that fictional case without checking it is a professional failure.That distinction matters.AI does not have professional duties of competence or candour.Lawyers do.
A generative AI system can produce:
- a case that never existed;
- a quotation that was never written;
- a real case with the wrong holding;
- a correct citation attached to an irrelevant proposition;
- or a mixture of real and fabricated authorities.
The lawyer’s responsibility begins precisely where the AI’s reliability ends.
Better AI Has Not Eliminated the Problem
There is a dangerous assumption that newer AI models have solved hallucination.They haven’t.The failure mode has simply become more sophisticated.
The obvious hallucination is easy to catch:
Smith v Jones, 999 F.3d 999 (2025).
A competent lawyer may immediately realise something is wrong.The more dangerous situation is a real case with a fabricated quotation.Or a real authority whose holding has been subtly misrepresented.Or a collection of legitimate authorities combined into an argument none of them actually supports.Those errors can look far more convincing.And therefore require more verification, not less.
Courts Are Starting to Build Verification Into Filing Rules
This is already visible in court practice.For example, the Eleventh Judicial Circuit in Miami-Dade County adopted an administrative order requiring lawyers and self-represented litigants who use generative AI in preparing certain court filings to disclose that use and certify that factual assertions, legal authority and citations have been independently reviewed and verified.
Other courts have adopted similar approaches.That is an important development.
The direction of travel is becoming clear:AI use may be permitted.Unverified AI work is not necessarily acceptable.
The distinction is crucial.
The Mandatory Citation Check
Every law firm using generative AI for legal research or drafting should have a simple rule:
No AI-generated legal authority reaches a client, court, regulator, or opposing counsel without independent verification.
That means checking:
Does the case exist?
Search the authoritative legal database.
Does the quotation exist?
Read the underlying judgment.
Does the case say what the AI claims?
Do not rely on the AI’s summary.
Is the authority still good law?
Check subsequent treatment and current status.
Does it actually support the proposition?
A real case can still be the wrong authority.
Is the jurisdiction correct?
A perfectly real decision from the wrong jurisdiction may be useless.
This should not be treated as optional quality control.
It should be part of the firm’s AI workflow.
The Human-in-the-Loop Is Not a Slogan
Many firms say:”AI is always subject to human oversight.”
But what does that actually mean?If a lawyer accepts an AI-generated citation without checking it, there was technically a human involved.There was not meaningful human oversight.
Human oversight means the person reviewing the AI output has:
- sufficient expertise;
- enough time;
- access to the underlying sources;
- authority to reject the output;
- and an obligation to independently assess its accuracy.
Otherwise, the human is functioning as a rubber stamp.
The Bigger Governance Problem
This is why AI hallucinations should not be treated solely as an AI training issue.They reveal weaknesses in the firm’s operating system.
Ask a firm:
Do you have an AI policy?
That is useful.
But ask instead:
What happens when a lawyer uses AI to draft a brief?
Who checks the citations?
Is verification mandatory?
Is it documented?
Which tools are approved?
What happens if an error is discovered after filing?
Who is accountable?
Those questions reveal whether governance actually exists.
The Risk Is Not Just a Bad Citation
A fabricated authority can have consequences far beyond embarrassment.
Depending on the circumstances, it can lead to:
- sanctions;
- fee awards;
- disciplinary referrals;
- reputational damage;
- loss of client confidence;
- wasted court time;
- adverse procedural consequences;
- and potentially much more serious professional consequences.
The Eleventh Circuit referral demonstrates that courts can move beyond correcting the filing and into questions of lawyer conduct.The CPS incident demonstrates the same principle from another direction: even where there was no deliberate attempt to mislead and the errors caused no impact on the court’s ultimate judgment, the failure to verify authorities was still identified as the operative human error.
The Lesson for Law Firm Leaders
The solution is not to ban AI.That would be an overly simplistic response.AI can dramatically improve legal research, drafting, document analysis and knowledge work.The solution is to design workflows in which AI’s strengths are used without transferring professional responsibility to the machine.
A mature workflow looks something like this:AI generates or identifies → Lawyer reviews → Source is independently verified → Lawyer exercises judgment → Final work is approved → Filing responsibility remains human.
That final step matters.The lawyer—not the model—signs the document.
AI Policy Should Become AI Procedure
A policy saying:
“Lawyers must verify AI-generated information.”
is better than nothing.But it is still incomplete.A serious firm should turn that principle into an actual procedure.
For example:
AI-generated authority identified
↓
Source independently located
↓
Citation checked
↓
Quotation checked against original
↓
Legal proposition independently assessed
↓
Human reviewer confirms accuracy
↓
Final document approved
That is governance you can actually operate.
The Most Dangerous AI Error Is the One That Gets Filed
An AI hallucination sitting inside a private conversation is a problem.An AI hallucination inside a court filing is something else entirely.The moment unverified output becomes an official legal submission, it enters a system where accuracy, candour and professional responsibility matter.That is why the critical control point is not necessarily the AI model.
It is the workflow between the model and the final filing.
Conclusion
The Eleventh Circuit case and the CPS incident are not evidence that lawyers should stop using AI.They are evidence that lawyers cannot stop checking AI.The technology will continue to improve.The hallucinations may become less frequent.The outputs may become more convincing.
None of that changes the fundamental rule:
The lawyer remains responsible for the work product.
The courts are increasingly making that expectation explicit.So if your firm’s AI policy does not include a mandatory verification process for citations, quotations, authorities and AI-generated legal analysis, the problem isn’t that your policy is slightly outdated.It may not be a real AI governance framework at all.AI can do the first draft.AI can accelerate the research.AI can surface the authority.
But before anything reaches a court:Check the source.Check the citation.Check the reasoning.And take responsibility for the result.
- AI Adoption Starts with Trust, Not Technology
- Why AI Adoption Fails in Law Firms: The Middle Management Bottleneck
- The Leadership Gap: Why AI Resistance at the Top Puts Law Firms at Risk
- The AI Copyright Reckoning: Who Owns Creativity in the Age of Generative AI?
- AI Hallucinations Are No Longer a New Problem. They Are Becoming a Professional Responsibility Problem.