lawpracticemanagement

Oops, I Did It Again: Lawyers Rely on AI at Your Peril

Lawyers continue to be misled by AI-generated case law that appears genuine but isn’t. This post is about a personal experience and why lawyers can’t afford to stop thinking.

I Gave GAI Clear Instructions: It Still Lied

A few weeks ago, for fun, I asked the GAI program I use to look on the internet and see if there was a quote on a specified topic from a “Founding Father”. Within seconds, it provided me with an on-point quote, which was attributed to John Adams, accompanied by a hyperlinked citation. It was the best party trick ever–until it wasn’t. Because the quote didn’t exist. Anywhere. When I called it out, GAI replied: “It sounds like something John Adams would say.”

Yesterday, I tested it again.

I asked for it to find the rule for a certain proposition. A rule of civil procedure that I knew existed. It told me the rule didn’t exist. I wanted to see if it would correct itself, so asked it to back that up with a case and a link to the statute. It did—with confidence. It even provided a quote from the case that it said supported the position it had taken. Except it was still wrong–the rule did exist and it had simply made up the quote.

When I pointed out the error and asked how this had happened, GAI explained:

“I incorrectly generalized and answered based on a commonly followed general rule.”

Mind you, I had given it specific, detailed instructions and prompts—things I had learned from CLE courses and articles about how to use AI and get accurate outputs. These included telling it not to make anything up, to double-check sources, and to provide links to public, official sources for every “fact” it retrieved from the internet.

What I got was a lie, wrapped in a polished, confident tone, dressed up like a real legal citation—because GAI is built to give me what I want and to sound persuasive and helpful, even when it’s dead wrong.

Lawyers’ Misuse of AI Continues to Make Headlines

Different courts, different lawyers, but the failure is identical: If you don’t read the case, the court will—and then you’ll make the news. Here is a partial list of headlines just from the past few weeks–hyperlinked to their source:

May 14, 2025 AI Hallucinations Strike Again: Two More Cases Where Lawyers Face Judicial Wrath for Fake Citations 

May 21, 2025, Judge Considers Sanctions Against Attorneys in Prison Case for Using AI in Court Filings

May 29, 2025, Lawyer Sanctioned $6,000 for AI-Generated Fake Legal Citations.

May 30, 2025, Southern District of Florida Sanctions Lawyers for Submission of AI Hallucinated Caselaw

May 31, 2025, US Lawyer Sanctioned After Being Caught Using ChatGPT for Court Brief

This Should Not be News to Most of Us

The problem of overworked lawyers attempting to take shortcuts is not new. Only the method has changed. For decades, lawyers have been getting sanctioned or called out by opposing counsel for:

  • Using the headnote from a paid online legal research tool as a “quote” without reading the opinion to confirm it.
  • Copying a pleading from a prior case and filing it without checking if the law still applies.
  • Lifting a motion from a CLE binder, online research tool, or lawyer listserv conversation and passing it off as their own.
  • Using the analysis from someone else’s case within the firm, without knowing or understanding the facts, court, or procedural history of that case. 

Every one of these examples has the same flaw: the lawyer wanted a way to circumvent doing the work we get paid to do i.e. think. 

The Real Problem Isn’t AI

AI isn’t the problem. It’s just the newest version of a long-standing temptation: to find a shortcut. Something to save time, make us look smart, or help us meet a deadline when the work hasn’t been done.

If you’re feeling pressure to use AI—or to do things faster, cheaper, or “more efficiently” than ever before—hear this:

You get paid to think, and no technology can replace your judgment or experience.

Your speed or formatting skills don’t determine your value. You are trained to analyze, reason, and argue. Your value lies in how you perceive what matters, identify what’s missing, and determine what it will take to achieve your client’s goals. You can’t delegate that to a machine just like you can’t outsource that to someone else’s pleading or form.

And don’t let fear push you to use a tool you don’t understand. Stop. Breathe. Learn what it can do. Learn what it can’t. Use it wisely—don’t rely on it to think for you, and don’t believe it when it assures you that it has. 

For Judges and Supervisors: A Fix Worth Considering

To stop this problem from recurring, consider this simple fix:

Require every pleading filed with the court that contains a reference, cite, or quotation to any authority to be internally hyperlinked to an attached appendix that includes a copy of the source with the relevant rule, holding, or quote highlighted for the court’s convenience.

This should become standard, just like a certificate of service. Lawyers should also apply this requirement to the work of those they supervise. And no, the clients should not pay for this “extra” work; it overhead–the price of doing business in the era of AI.

The Technology Changed, the Job Didn’t

This isn’t about shaming lawyers. It’s about reminding us who we are.

We are not prompt engineers or data processors. We are professionals who took an oath and have duties to our clients, the courts, and the public.

So please, don’t be a headline. 

Read the case. Check the quotes. Confirm the law is still good. And don’t rely on any tool that doesn’t distinguish between the truth and a lie.

Generative AI for Lawyers Part 3: Ethical Use of Hypotheticals with GAI

By Jeanne M. Huey

In the previous entry in this series, we discussed ABA Formal Opinion 512’s admonition against inputting any information relating to a client’s representation (confidential information under ABA Model Rule 1.6) into self-learning generative artificial intelligence (GAI) software without first obtaining the client’s “informed consent”.

Consider, however, that there are a variety of ways to use self-learning GAI without either disclosing any client confidential information or obtaining the client’s “informed consent.” These methods allow you to reap the benefits of self-learning GAI for your clients while maintaining your ethical obligations under the relevant rules and opinions.

Ethical Use of Hypotheticals in Case Discussions

ABA Model Rule 1.6 Comment 4 advises that using a hypothetical to “discuss issues relating to the representation is permissible so long as there is no reasonable likelihood that the listener will be able to ascertain the identity of the client or the situation involved.” In any public-facing discussion, these prohibitions are particularly relevant, because audience members can easily connect the dots from contextual information in hypotheticals to learn who the client is or simply look up the speaker’s cases online to get the same information. This makes it nearly impossible to say that there is “no reasonable likelihood” that client confidential information will be disclosed when lawyers use hypotheticals based on their clients and cases in situations where the lawyer’s identity is known.

ABA Formal Opinion 480 (public comments by lawyers such as blog posts and social media) and ABA Formal Opinion 511R (lawyers discussing cases on lawyer forums such as LISTSERV) take this principle further. Both opinions warn against the use of hypotheticals in any public context without the client’s informed consent when there’s a possibility that third parties could deduce the identity or specific details of a client’s situation.

The combined guidance from these ethics opinions and related rules establish a high bar for confidentiality in public legal commentary and make it clear that clever wordsmithing and linguistic acrobatics alone (i.e., using hypotheticals) will not insulate lawyers from the risk of liability for a breach of the rules regarding confidentiality in these kinds of settings.

Advanced Strategies for Securely Using Hypotheticals with GAI

While any use of a hypothetical to discuss a client or case requires vigilance regarding client confidentiality, GAI may offer a more protective environment for the use of a well-written hypothetical.

In contrast to using hypotheticals in public discussions or commentary, using a hypothetical in a GAI prompt significantly reduces the likelihood of associating a scenario with a specific client due to GAI’s inherent anonymity; there is no identifiable author or audience involved. This allows attorneys to use hypotheticals to effectively analyze complex legal issues while safeguarding client confidentiality and to do so without seeking informed consent from the client. This distinction aligns with ABA guidelines on confidentiality, which emphasize that the risk of disclosure varies depending on the audience and platform used.

Consider the following approaches:

1.     Abstract the Core Legal Issue

When crafting hypotheticals, distill the client’s situation to its fundamental legal principles, omitting any specifics that could reveal their identity—such as names, dates, financial figures, or unique circumstances. Focus on the legal doctrines or statutory interpretations at play rather than granular details.

Example:

Instead of asking about a “global pharmaceutical company facing allegations of off-label marketing in violation of FDA regulations,” reframe the inquiry to address “a corporation navigating regulatory compliance challenges in the context of strict federal oversight.” This allows you to explore the complexities of regulatory compliance without disclosing identifiable information.

2.     Emphasize Legal Theories and Precedents

Center your inquiries on broader legal theories, jurisprudence, or procedural rules. You can gain insights without tying the discussion to specific client facts by focusing on legal frameworks and landmark cases.

Example:

Rather than delving into the specifics of a complex international arbitration your client is involved in, you might ask, “How have recent court decisions impacted the enforcement of arbitral awards under the New York Convention?” This approach examines critical legal interpretations without disclosing client involvement.

3.     Frame Questions Around Legal Processes and Strategies

Pose questions that concentrate on legal processes, litigation strategies, or best practices rather than specific factual scenarios. This enables you to gather strategic insights while maintaining confidentiality.

Example:

If advising a client on mitigating risks in cross-border mergers, you could ask, “What are effective due diligence strategies for uncovering potential liabilities in international M&A transactions?” This allows exploration of procedural tactics without referencing the client’s situation.

4.    Use Analogous Legal Contexts

Construct hypotheticals using different contexts or industries that parallel your client’s issues. By transferring the legal problem to a comparable but distinct setting, you can examine relevant principles while further obscuring client details.

Example:

If your client is an energy company facing environmental litigation over alleged contamination, you might frame the hypothetical around a manufacturing company dealing with toxic-tort claims. The legal principles regarding environmental liability and defense strategies remain pertinent, but the context shift protects client confidentiality.

By employing these techniques, you can effectively leverage GAI tools to explore complex legal issues while rigorously safeguarding client confidentiality.

© 2024 by the American Bar Association. Reproduced with permission. All rights reserved. This information or any portion thereof may not be copied or disseminated in any form or by any means or stored in an electronic database or retrieval system without the express written consent of the American Bar Association.

Generative AI for Lawyers Part 2: Maintaining Confidentiality

ABA Formal Op. 512 focuses on the risks of using generative AI (GAI) in legal practice, with a key concern being the confidentiality of client information. Under ABA Model Rule 1.6, lawyers are obligated to protect all client-related information, including preventing inadvertent or unauthorized access. ABA Model Rule 1.9(c) extends this duty to former clients, and ABA Model Rule 1.18(b) to prospective clients.

Unauthorized Disclosure of Confidential Information: What Is the Risk with GAI?

Self-learning GAI poses a higher risk to client confidentiality than other technology used in a modern law practice because it can retain and reuse input data (prompts), increasing the chance of inadvertent disclosure or cross-use in other cases. This is true whether the information is used within a firm’s closed system—where the stored data is only used internally—or outside the firm in an open system—where data is shared with external sources.

Why do lawyers need to be concerned about inputting confidential information into an internal firm or “closed” GAI system? The answer lies in the distinction between access to confidential information and the use of that information within a firm. While lawyers and staff typically have access to all of the firm’s clients’ confidential information, using that information to prompt the firm’s self-learning GAI system creates a real risk that one client’s information may be applied to other clients’ cases. This may result in a breach of the confidentiality obligations owed to the first client and could occur without either lawyer realizing that a violation has taken place.

This risk is not just hypothetical. Multiple ethics opinions, including Opinion 512 and those issued by the Florida Bar and Pennsylvania & Philadelphia Bars, emphasize that self- learning GAI tools may inadvertently cause the disclosure of client information even in a closed system used exclusively within a single law firm.

Informed Consent—A Prerequisite for Using Confidential Information with GAI

So, what can be done to avoid a violation of Model Rule 1.6 for unauthorized disclosure of confidential client information under these circumstances? Opinion 512 concludes that, due to the unique risks posed by self-learning GAI, lawyers should obtain “informed consent” from the client before using any information related to the representation in GAI prompts—even within a firm’s “closed system.”

The opinion is quick to note that informed consent cannot be accomplished by a boilerplate acknowledgment or notice clause in an engagement letter. Informed consent is a defined term in ABA Model Rule 1.0(e) and requires that the lawyer provide the client with “adequate information and explanation about the material risks of and reasonably available alternatives to” the proposed conduct.

Opinion 512 explains that “adequate information and explanation” under these conditions calls for a “meaningful dialogue” with the client that includes:

  • the lawyer’s best judgment about why the GAI tool is being used;
  • the extent of and specific information about the risks involved in disclosing client information;
  • particulars about the kinds of client information that will be disclosed;
  • the ways in which others might use the information against the client’s interests;
  • a clear explanation of the GAI tool’s benefits to the representation; and
  • the risk that later users or beneficiaries of the GAI tool will have access to information relating to the representation.

This list, from Opinion 512, makes it clear that any lawyer seeking informed consent must have more than a general awareness of GAI technology. They must, as ABA Model Rule 1.1 Comment 8 sets out, be competent in understanding the benefits and risks of that technology.

Obtaining informed consent here aligns with a lawyer’s duty to communicate effectively with their client about the work being undertaken in their case. Under ABA Model Rule 1.4, lawyers must inform clients about decisions affecting their representation, including the means proposed to achieve the client’s objectives. When using client confidential information in a self-learning GAI system is proposed, the client must be given enough information to make an informed decision about whether to permit it.

Finally, while “informed consent” does not require written consent, the best practice is to confirm the client’s consent either 1) in a writing by the client or 2) from the lawyer confirming the client’s oral consent. (See ABA Model Rule 1.0(b)).This approach helps protect the client’s interests and discharge the lawyer’s ethical duties, ensuring that trust and transparency remain intact throughout the representation.

In Part 3 of this series, we will explore how you can use self-learning GAI tools to benefit your client without disclosing information about the representation (confidential information) or obtaining informed consent.

© 2024 by the American Bar Association. Reproduced with permission. All rights reserved. This information or any portion thereof may not be copied or disseminated in any form or by any means or stored in an electronic database or retrieval system without the express written consent of the American Bar Association.