How can Legal use AI without losing critical thinking?

Written by Summize CEO Tom Dunlop

Our latest AI Fluency Report found something that probably won’t come as a surprise – 100% of legal professionals surveyed are using AI, 91% rate their understanding of generative AI as advanced or expert, while 96% are confident they can spot an AI response that’s inaccurate, incomplete or misleading.

Those numbers tell us that legal professionals aren’t approaching AI naively. They’re using it, understanding it and by their own assessment, feeling confident in their ability to challenge it.

But there’s a more interesting question underneath those numbers: how do you know when your confidence is justified?

Because being confident using AI and being confident in what AI tells you are two different things.

Is it harder to check AI if you rely on it for knowledge?

It’s easy to assume every lawyer is verifying every AI response, but our research found that only 30% of respondents verify against a primary source when an AI response doesn’t look right. Only 11% would discard the response entirely.

A lawyer’s role is advisory by nature. You draw on your own experience, research legislation and precedent, understand what’s happening in the business and then make a judgment call. AI is a natural fit for parts of that process. It can help generate documents, summarize information and get you to a useful first draft much faster. But there’s an important distinction in how you use it.

If AI is helping you produce something you already understand, you’re in a strong position to assess the result. If it generates a contract clause, for example, you can usually tell quite quickly whether it looks right. You’ve drafted clauses before. You’ve negotiated them. You know what good looks like and where the edge cases sit.

That’s very different from asking AI to tell you something you don’t already know.

If you’re relying on AI to research an unfamiliar regulatory issue, explain a novel legal question or inform a high-value decision, you’re relying on it for the knowledge itself. That makes the output harder to critique, because you don’t necessarily have the underlying knowledge needed to recognize when something is wrong. The more you rely on AI to know something for you, the less equipped you may be to judge whether it’s right.

That’s where critical thinking becomes more important than confidence with the tool.

The real cost of unchecked AI is the verification tax

This creates a problem if you’re trying to capture AI’s productivity benefits. AI is supposed to remove work – but if you’re asking it to generate an answer without giving it the context needed to make that answer reliable, you can end up moving the work rather than removing it.  

The more you have to check, the less meaningful the original time-saving becomes. But “verify everything” isn’t a workable standard, especially as AI becomes more and more widely used. Instead, the approach needs to reflect the task.

AI could always be wrong – the bigger question is whether you can tell it’s wrong. If the answer is yes, you can make a more informed judgment about how much scrutiny the output needs. If the answer is no, that’s a signal to go back to underlying source material.

When should lawyers verify an AI-generated answer?

The right level of scrutiny depends on three things:

1. How much you already know

2. How much you’re relying on AI’s own knowledge

3. What happens if the answer is wrong.

For lower-risk work based on knowledge you already have, a lighter check may be appropriate. An internal advice memo, board note or summary of a contract you’ve already reviewed can be useful applications of AI. If you’ve provided the relevant information and context yourself, you’ll have a strong sense of whether the output makes sense.

Higher-risk work needs a different instinct. Legal conclusions, regulatory advice, contractual positions and decisions with significant business consequences deserve greater scrutiny, particularly when the subject is outside your direct experience. If you’re using AI to inform a decision about an unfamiliar tax structure, for example, that’s a point where going back to the primary source matters. You don’t want an answer that sounds plausible because AI has merged several related concepts together. You need to know what the underlying authority actually says.

The line is about how much you know, and how much is riding on being right, which is more useful than simply saying “lawyers should check everything.”

Can better context reduce the verification burden?

Verification shouldn’t cancel out the productivity gains AI is supposed to deliver, but we’re in danger of that happening if the underlying knowledge isn’t structured properly. If I’m relying on AI for context and just using it to produce everything without validating it against my contracts, my playbooks or policies, I end up double-checking every line, because it hasn’t got that built in context. At that point it’s more manual effort than doing it myself.

The fix isn’t more manual checking, but better inputs in the first place. That means treating your knowledge in layers – regulations and statutes, your contracts, your policy documents, your internal governance – and structuring that knowledge so it’s accessible and easy to plug straight into an AI model.  

For example, you could structure your knowledge by:

• Contracts – capturing consistent information such as parties, key dates, obligations, termination rights and commercial terms

• Playbooks – separating preferred positions from fallback positions, approval thresholds and escalation points

• Policies – making relevant rules, exceptions and owners clear and easy to reference

• Regulations and legislation – keeping the source, jurisdiction, effective date and relevant provisions clear so AI can distinguish current requirements from outdated or irrelevant material

• Internal governance – documenting who needs to approve what, which teams need to be involved and what happens when a request falls outside the usual process

Get that foundation right, and you’re verifying against a trusted context database. The checking gets faster because the starting point is already closer to correct.

How do you know what to trust AI with?

There’s a difference between being AI confident and AI fluent. Confidence is knowing how to use the tools: prompting them effectively, incorporating them into everyday work and assessing whether an output looks useful.

Fluency goes further. It’s understanding what you’re asking AI to do, what knowledge it’s working from and, what assumptions sit behind the answer and when you need to go back to a primary source.

When used well, AI should create more room for critical thinking, not less – and because of that, lawyers will be able to spend less time on the groundwork. Machines can accelerate that, and their expertise can go towards assessing, challenging and making the decisions that ultimately matter. But that only works when AI has access to the right knowledge and context in the first place.

That’s where a contract knowledge layer can help. By bringing your contracts, policies and business knowledge together, it gives AI access to the information your legal team already relies on – helping lawyers spend less time checking and more time applying their expertise.