When AI “hallucinates” in court: a wake-up call for the legal profession
The recent incident involving Sullivan & Cromwell is not just another AI mishap but a stark reminder of what is at stake when technology is used without professional discipline.
What happened?
In April 2026, one of the world’s most elite law firms apologised to a U.S. bankruptcy judge after submitting a court filing containing AI-generated inaccuracies (so-called “hallucinations”). These included fabricated case citations and misstatements of law.
The most striking part?
The errors were not caught internally.
They were identified by opposing counsel.
The firm admitted that its own AI policies and review processes were not followed.
A corrected filing was later submitted—but the damage, reputationally and procedurally, had already been done.
The real issue: not AI, but uncritical reliance
Courts are seeing a growing number of similar incidents:
Lawyers have already been sanctioned for relying on AI-generated fake authorities.
Databases now track hundreds of AI-related errors in legal filings.
We now see a pattern. And let’s be clear: AI is not at fault, but those who use it without proper critical verification are. It’s like blaming a junior for making mistakes when no supervision has been provided (depending on the type of mistake ofc).
Large language models are not reasoning systems; they are prediction engines. They optimise for coherence, not truth. That means they can produce outputs that look authoritative, structured, and legally plausible—yet are entirely wrong. In other words: errors that look like expertise.
Let’s remember that legal work is not just about drafting. It is about accuracy, accountability, and reliance. A court filing is not a draft blog post. Submitting incorrect authorities is more than a technical glitch; it undermines credibility, professional integrity, and the administration of justice. And crucially, the responsibility never shifts to the tool.
Under the SRA Code of Conduct, solicitors must:
maintain competence
provide services with skill and care
ensure work is properly supervised and checked
Using AI without verification is a potential breach of professional duties. Competence in 2026, therefore, includes something new but fundamental:
The ability to critically assess AI-generated output.
Not just to use tools but to challenge them.
The profession is clearly moving towards a new baseline of AI literacy. This incident should not discourage the use of AI in legal practice. It should sharpen it. Because the future lawyer is not the one who avoids AI but one who knows how to use it critically.
You can find more about the incident here.
