We've seen repeated issues where lawyers and others have been caught failing to cite check their work (as discussed here and here and here, for example). In what might be a first, a judge has relied on a submission including support that doesn't exist and was likely hallucinated by generative AI. A Georgia appellate court has vacated a divorce judgment after discovering that the trial court’s order relied on fictitious case law from submissions prepared by the husband’s attorney. The fake cases were used to deny the wife’s motion to reopen the case, which challenged the validity of service by publication.

What set this case apart was not only that the husband's filings included numerous hallucinated citations, but that the trial court adopted two of those bogus cases verbatim into its order without scrutiny. The appeals court expressed concern that the trial judge failed to recognize the citations as fictitious, noting that the defect was “apparent on [the] face” of the order and undermined the presumption that trial courts follow the law. Because of this fundamental defect, the appellate court found it could not conduct meaningful review and ordered a new hearing.

The appellate court did not make factual findings about whether the husband's attorney personally fabricated the cases or used generative AI tools. But it cited rising judicial concern over unverified AI-generated legal content (referencing Chief Justice Roberts's 2023 report highlighting the need for caution) and imposed a $2,500 sanction against the attorney for filing a frivolous motion for attorneys' fees, which itself cited a fake case.

This has gone beyond being a “cautionary tale”—the limitations and risks inherent in using AI for research and drafting are well-known. The fact that AI-hallucinated precedent is still making its way into filings, let alone rulings, should shock us all. The reliance on unverified, fictitious citations undermines the integrity of our legal system and diminishes the quality of judicial decision-making. We all must do better.