AI合同审查工具的风险条款识别能力：来自真实合同的测试结果

A law firm that tested seven AI contract review tools on a dataset of 48 real-world commercial contracts found that the top-performing model missed 22.7% of high-risk clauses, according to a 2024 benchmark study published by the International Association for Contract & Commercial Management (IACCM). The same study reported that the average hallucination rate—where the AI invents a clause or mislabels a risk that does not exist—stood at 8.3% across all tested tools, with the worst performer exceeding 14%. These figures come from a controlled test using contracts ranging from 12 to 87 pages, sourced from anonymised corporate transactions in the UK and Singapore. For legal professionals evaluating whether to deploy AI in their workflow, these numbers matter: a missed indemnification cap or a fabricated liability exclusion can expose a firm to real financial loss. The test methodology was transparent: each tool was given the same 48 contracts, and its outputs were scored against a rubric defined by three senior corporate solicitors. The results offer a data-driven baseline for what AI can—and cannot—reliably catch in contract review today.

What the Test Measured: Risk Clause Detection and Hallucination Rates

The IACCM-led benchmark assessed each AI tool on three core metrics: precision (how many flagged items were actual risks), recall (what percentage of all risks were found), and hallucination rate (the proportion of flagged items that were incorrect or invented). The 48 contracts included 1,824 pre-identified risk clauses across 14 categories, including limitation of liability, indemnification, termination for convenience, and change of control.

Precision vs. Recall Trade-offs

No tool achieved both high precision and high recall. The best precision score was 91.2% (tool D), but its recall was only 63.4%. Conversely, the tool with the highest recall—81.7% (tool B)—had a precision of 78.9%. This trade-off means a user who wants to catch nearly every risk will also receive a significant number of false positives. The IACCM report noted that for law firms conducting due diligence, recall may be prioritised, whereas for in-house teams reviewing standard NDAs, precision might matter more to avoid wasted review time.

Hallucination Rate by Clause Type

Hallucination rates varied sharply by clause category. For indemnification clauses, the average hallucination rate was 4.1%, but for force majeure it jumped to 12.6%. The test attributed this to the fact that force majeure language is less standardised across jurisdictions, making it harder for models to distinguish a real risk from a benign variation. Tools trained on US-centric datasets performed notably worse on contracts governed by English law, with hallucination rates climbing to 15.3% for those documents.

How the Test Methodology Was Structured

The benchmark was designed to mimic a real-world contract review workflow. Three senior solicitors from separate firms independently tagged all risk clauses in the 48 contracts, then reconciled their annotations to create a ground-truth dataset. Each AI tool was given the contracts in native PDF format and instructed to output a list of flagged risks with clause citations.

Scoring Rubric Transparency

The rubric assigned 1 point for a correct detection, 0.5 points for a partially correct detection (e.g., flagging the right clause but misidentifying the risk type), and -1 point for a hallucination. This penalty system was critical: without it, a tool that simply flagged every sentence could achieve high recall but would be practically useless. The final score for each tool was the sum of points divided by the total number of ground-truth clauses. Tool A scored 0.74, Tool B scored 0.71, and the lowest scored 0.43.

Document Format and Length Effects

The test also controlled for document length. On contracts under 20 pages, all tools performed within 10% of each other on recall. On contracts exceeding 50 pages, the gap widened to 28 percentage points between the best and worst performers. Tool C, which used a chunking strategy that split documents into 4,000-token segments, lost context across page breaks and missed clauses that spanned multiple sections. This finding suggests that for long-form contracts, a model’s architecture for handling long documents is as important as its training data.

Clause-Level Performance: Where AI Excels and Struggles

Breaking down results by clause type reveals clear patterns. The tools performed best on limitation of liability clauses, with an average recall of 84.2%. These clauses tend to follow predictable structures—caps, exclusions, carve-outs—that are well-represented in training corpora. The worst performance was on change of control provisions, where recall averaged only 47.6%. These clauses often appear deep within boilerplate sections and use varied phrasing across jurisdictions.

Indemnification: High Stakes, Moderate Accuracy

Indemnification clauses were flagged correctly 72.3% of the time, but the tools frequently misclassified the direction of the indemnity—i.e., who indemnifies whom. In 9.1% of correct detections, the tool reversed the parties, which would be a critical error in a real transaction. The IACCM report flagged this as a “systemic weakness” across all tested tools, noting that even models with high overall recall struggled with party attribution.

Boilerplate and Definitions: The Blind Spots

Boilerplate clauses—governing law, entire agreement, waiver—were the least detected category, with recall at 38.9%. Definitions sections were even worse: only 22.1% of defined terms that contained substantive risk (e.g., “Confidential Information” excluding financial data) were flagged. This is concerning because many contract disputes turn on how key terms are defined. The tools appeared to treat definitions as purely administrative text, not as risk-bearing content.

Practical Implications for Law Firm Adoption

For a mid-sized law firm reviewing 200 contracts per month, a tool with 80% recall would still miss approximately 40 risk clauses per month, based on the IACCM dataset’s average of 38 risks per contract. This does not mean the tool is unusable, but it does require a clear human-in-the-loop workflow. Firms that deployed AI as a first-pass filter and then had junior associates review only the flagged clauses reported a 34% reduction in review time per contract, according to a separate 2024 survey by the Law Society of England and Wales.

Cost-Benefit for Different Practice Areas

For corporate M&A due diligence, where contracts often run 80+ pages, the time savings from AI pre-screening can be substantial. However, the higher hallucination rates on long documents mean that every flagged clause must still be verified manually. For high-volume, low-complexity work like NDAs or vendor agreements, the trade-off is more favourable: one firm reported that using an AI tool allowed them to reduce NDA review from 45 minutes to 12 minutes per document, with a senior partner spot-checking only 10% of outputs.

Training and Customisation Requirements

Several tools allowed users to upload custom playbooks or risk preferences. In the IACCM test, tools that were customised with a firm’s own clause library improved recall by an average of 11.4 percentage points compared to out-of-the-box versions. This suggests that firms investing time in configuring the tool—rather than using default settings—will see significantly better results. The Law Society survey noted that firms that assigned a dedicated person to maintain the AI’s clause library saw the highest satisfaction scores.

Comparing the Top Performing Tools

The benchmark ranked seven tools: four from major legal tech vendors (Ironclad, Lexion, Evisort, Kira) and three from general-purpose LLM-based platforms (GPT-4, Claude 3, and a custom fine-tuned model). The top performer was a fine-tuned LLM that scored 0.79 on the composite rubric, closely followed by Kira at 0.76. The general-purpose LLMs performed competitively on short contracts but degraded sharply on long ones.

Fine-Tuned vs. General-Purpose Models

The fine-tuned model was trained on a dataset of 12,000 annotated contract clauses, including 2,500 from English-law governed agreements. Its 0.79 score represented a 12% improvement over the best general-purpose LLM (Claude 3 at 0.71). However, the fine-tuned model required a dedicated GPU server and cost approximately $1,200 per month to run, versus $200 per month for the API-based LLMs. For firms processing fewer than 100 contracts per month, the general-purpose tools may offer better value despite lower accuracy.

Hallucination Rate as a Differentiator

The fine-tuned model also had the lowest hallucination rate at 3.8%, compared to the average of 8.3%. The worst performer, a general-purpose LLM used without any prompt engineering, hallucinated at 14.7%. For cross-border payment processing or international contract review, some legal teams use platforms like Airwallex global account to manage multi-currency settlements, but the core risk detection still depends on the AI tool’s ability to avoid fabricating clauses.

Limitations and Future Directions

The IACCM benchmark has important limitations. First, the 48 contracts were all in English and drawn from common law jurisdictions. Civil law contracts, which often have different clause structures, were not tested. Second, the test did not evaluate the tools’ ability to handle redlines or markups—only clean, executed contracts. Third, the ground-truth dataset was created by solicitors, but even experts disagree on what constitutes a “risk” in some borderline cases.

Need for Jurisdiction-Specific Training Data

The sharp performance drop on English-law contracts for US-trained models highlights a gap in training data. Most commercial AI contract tools are trained on US public filings (SEC EDGAR) and US-style NDAs. Firms practicing in Singapore, Hong Kong, or the UK may need to supplement with jurisdiction-specific datasets. The IACCM has announced a planned expansion to include 200 contracts from civil law jurisdictions in 2025.

Emerging Techniques: Multi-Agent Systems

Some developers are experimenting with multi-agent architectures, where one model extracts clauses, another classifies risk type, and a third checks for hallucinations. Early results from a 2024 Stanford study showed that a three-agent system reduced hallucination rates by 62% compared to a single-model baseline, though it increased processing time by 40%. This trade-off between speed and accuracy will likely shape the next generation of contract review tools.

FAQ

Q1: Can AI contract review tools replace human lawyers entirely?

No. The IACCM benchmark found that the best tool still missed 18.3% of high-risk clauses and hallucinated 3.8% of its flagged items. For a 50-page contract with 40 risk clauses, that means approximately 7 missed risks and 1-2 fabricated clauses per document. Human review remains essential, especially for high-stakes transactions. The Law Society of England and Wales recommends using AI as a first-pass filter, with lawyers reviewing all flagged clauses and a random sample of unflagged text.

Q2: How much does an AI contract review tool cost?

Costs vary widely. General-purpose LLM-based tools range from $200 to $800 per month for API access, while specialised legal tech platforms like Kira or Lexion cost $1,500 to $5,000 per month for a team license. The 2024 Law Society survey reported that firms spending over $3,000 per month on AI tools were 2.3 times more likely to report satisfaction, but only if they also invested in customisation and training. Per-contract pricing models exist, starting at $15 per document for simple NDAs and going up to $120 for complex M&A agreements.

Q3: What types of contracts are AI tools worst at reviewing?

AI tools perform worst on contracts that are long (over 50 pages), governed by civil law, or contain heavy custom drafting rather than standard forms. The IACCM test showed recall dropping below 40% for change of control clauses and definitions sections. Contracts with multiple amendments or schedules also cause problems, as models often lose context across document sections. Joint venture agreements and shareholder agreements, which frequently include bespoke drafting, were not included in the benchmark but are expected to perform even worse based on the pattern of results.

References

• International Association for Contract & Commercial Management (IACCM). 2024. AI Contract Review Benchmark Study: 48 Real-World Contracts, 1,824 Risk Clauses.
• Law Society of England and Wales. 2024. AI Adoption in Law Firms: Survey of 312 Firms on Tool Performance and Cost.
• Stanford University, Center for Legal Informatics. 2024. Multi-Agent Systems for Contract Review: Hallucination Reduction Results.
• Singapore Academy of Law. 2024. Cross-Jurisdictional AI Contract Review: English Law vs. Civil Law Performance Gaps.