Test the update path, not just the first answer

A litigation system that works only on a static upload has missed a defining characteristic of litigation.

Cases change.

A new pathology report arrives. A supplemental interrogatory response changes the damages theory. A deposition conflicts with the complaint. An amended pleading introduces a new allegation. A newly decided case changes the legal framework. An expert report makes a previously minor factual dispute strategically important.

Ask the vendor to show what happens next.

Does the system rebuild everything from scratch? Does it append new information without reconciling it against what came before? Does it know which analyses may now be stale? Can it identify which earlier conclusions should be reconsidered?

This is one reason we treat the Fact Graph as dynamic. The graph for a medical malpractice matter should not have the same shape as one for construction, employment, trucking, or product liability. Even within one practice area, two matters should not be forced into an identical map.

As the evidentiary record changes, the representation of the case should change with it.

A useful diligence exercise is to run an initial review, add material records, and rerun the relevant workflow. Do not tell the vendor what changed. See whether the system detects it.

Force the system to confront contradictions

A system that only summarizes what it finds is doing document processing. Litigation requires more.

Defense work turns on disagreement:

• allegation versus record;

• complaint versus deposition;

• witness versus witness;

• prior medical history versus claimed causation;

• contemporaneous note versus later recollection;

• expert assumption versus underlying data;

• discovery response versus documentary evidence;

• one authority versus another.

Your test set should deliberately contain contradictions.

Then ask the system questions whose correct answer cannot be produced by retrieving one convenient passage.

A useful system should distinguish between established facts, allegations, disputed facts, inferences, and missing evidence. It should be able to surface competing sources rather than silently selecting one version.

Also test absence.

Ask a question for which the record does not yet support a confident answer. The system should make the gap visible. A confident sentence unsupported by the matter file is not case intelligence.

Understand the privilege and confidentiality posture

Security questionnaires are necessary. They are not sufficient.

Law firms should understand what actually happens to client information after it enters a system.

ABA Formal Opinion 512 instructs lawyers to consider how a provider protects confidentiality, its security measures and policies, whether information is retained, and the risks associated with third-party systems handling information relating to the representation.

American Bar Association

Florida Ethics Opinion 24-1 makes the issue especially concrete. It tells lawyers evaluating generative AI to investigate policies concerning data retention, data sharing, and self-learning, while emphasizing that lawyers remain responsible for professional judgment and the verification of AI-assisted work. The Florida opinion is advisory and jurisdiction-specific, but its diligence questions are useful well beyond Florida.

The Florida Bar

Ask plainly:

• Is our client information used to train a shared or public model?

• Can information from one client or matter influence results for another?

• What information is retained, where, and for how long?

• What happens to copies, indexes, embeddings, logs, and backups when a matter is deleted?

• Which subprocessors can access the data?

• How are permissions and ethical walls enforced?

• Can vendor personnel access matter content, and under what controls?

• What happens when a user loses access to a matter in the firm's source system?

• What contractual commitments govern these answers?

The buyer should understand the architecture well enough to explain the risk internally. "Enterprise grade" is not an architecture.

Put human gates where consequences increase

Not every AI-assisted task carries the same risk.

A first-pass internal chronology is different from a pleading ready for filing. An internal list of possible deposition topics is different from a settlement recommendation sent to a carrier. A draft research path is different from a representation to a court.

NCSC guidance recommends increasing human involvement as the risk of the use case increases and emphasizes that AI-generated content should be reviewed for accuracy.

National Center for State Courts

Buyers should ask where those gates exist in the product.

Can an AI-generated conclusion flow directly into an external deliverable without review? Can the reviewer inspect the evidence before accepting it? Is it clear what the system generated versus what a lawyer approved? Can a partner revise the analysis without fighting the software?

The best human-in-the-loop design does not merely put an approval button at the end. It makes professional review easier throughout the workflow.

Separate data isolation from model branding

A vendor can use a sophisticated model and still have a poor data-governance architecture.

The diligence question is not simply which underlying model provider is involved. Ask what information is sent to each model, under what contractual terms, with what retention settings, through which infrastructure, and whether customer information is used for training.

Then distinguish several different concepts that are often compressed into one sales answer:

• Training. Is client information used to train or improve any shared model?

• Inference. What information is transmitted when the system processes a request?

• Storage. What information does the application itself retain?

• Isolation. What prevents one firm's or client's information from becoming accessible to another?

• Human access. Under what circumstances can vendor personnel inspect customer content?

"Your data is private" is a conclusion. Procurement needs the underlying controls.

Ask for compliance evidence, not compliance vocabulary

Security diligence should produce artifacts.

Depending on the firm's requirements and the sensitivity of the work, buyers may need to review materials such as independent assurance reports, penetration-test summaries, security architecture, data-flow documentation, incident-response procedures, subprocessor information, encryption practices, access controls, business continuity procedures, retention policies, and contractual security commitments.

Do not reduce the exercise to whether a logo appears on a security page.

Ask what the artifact covers.

Which product environment was assessed? Which period? Are there exceptions? Are customer responsibilities identified? Has anything material changed since the assessment? Does the vendor's actual proposed deployment fall within the audited environment?

Carriers should apply the same discipline when a platform will touch panel-firm matters or claim information.

Compliance should narrow uncertainty. It should not substitute for understanding the system.

Demand honesty about the DMS

Most firms already have a system of record.

An AI platform should be able to explain its relationship to that system without hand waving.

Turbo Law, for example, does not position itself as a replacement for a firm's DMS. The DMS remains the system of record, while the litigation platform turns matter information into connected case intelligence and work product.

Turbow Law

Whatever platform you are evaluating, ask:

• What exactly is synchronized?

• Are documents copied, indexed, referenced, or all three?

• How are document versions handled?

• How quickly do permission changes propagate?

• What happens if a source document is replaced?

• Can the platform identify the authoritative version?

• Does deletion in the DMS affect data downstream?

• What metadata comes across?

• Does the system write anything back?

• What part of the promised integration is available today?

That final question matters.

If the pilot is running through manual uploads while the enterprise case depends on a deep DMS integration, evaluate those as two different things. A roadmap is not a production integration.

Design the pilot to disprove the product

Do not run a pilot whose purpose is to generate a positive testimonial.

Try to break the system.

Use the firm's own files, subject to appropriate confidentiality and security review. Select matters that actually represent the work you expect the platform to handle. Include messy productions, inconsistent naming, duplicate documents, amended pleadings, incomplete records, and contradictions.

We recommend testing several distinct workflows rather than one generic chat exercise. For example:

Intake or early case assessment: Can the system turn the initial file into a cited view of allegations, key facts, exposure questions, defenses, and information gaps?

Record analysis: Can it build a chronology or issue map across heterogeneous documents and open important propositions to their sources?

Later-stage work: Can that matter intelligence support a discovery draft, motion analysis, strategy update, or client report after the record changes?

Then introduce new evidence.

See what updates. See what does not.

Measure the things that affect professional work: unsupported propositions, missed material facts, incorrect citations, reviewer corrections, contradiction detection, time required to verify important claims, consistency after updates, and whether the final work product can actually be used by the responsible lawyer.

Set the firm's pass/fail criteria before the pilot begins.

Use this diligence checklist before signing

A serious evaluation should leave the buyer able to answer these questions:

• Job: Which litigation workflows are we buying this for?

• Matter unit: What representation of the case persists across those workflows?

• Sources: Can important factual and legal propositions be opened directly to authoritative source material?

• Updates: What happens when pleadings, discovery, records, testimony, or law change?

• Contradictions: Does the system surface conflicting evidence and competing propositions?

• Uncertainty: Does it reveal gaps, or manufacture certainty?

• Privilege: How are client confidentiality, ethical walls, and permissions handled?

• Human review: Where are verification and decision gates built into the workflow?

• Training: Is client information used to train or improve shared models?

• Isolation: How is one matter, client, or firm prevented from contaminating another?

• Security evidence: Which claims are supported by actual compliance and security artifacts?

• DMS: What is live today, and what remains a planned integration?

• Pilot: Has the product been tested on representative firm files under predefined success criteria?

If the vendor cannot answer those questions clearly, another model benchmark will not fix the problem.

Category names are cheap. Method is what matters.

The legal technology market will use terms like "case intelligence", "matter intelligence", "agentic", "knowledge graph", and "Fact Graph".

None of those labels should earn procurement credit by themselves.

The buyer should ask what the system actually represents, how that representation changes, how it connects to evidence, how contradictions are handled, and how that intelligence carries into subsequent work.

That standard applies to us too.

When we describe Turbo Law's Fact Graph, the meaningful claim is not that we use a graph. It is that the graph is intended to capture a changing understanding of the individual litigation matter, beyond extracted entities, and connect pleadings, discovery, case law, and case specifics so that review, drafting, research, strategy, and reporting operate from the same matter intelligence. Turbo Law also hyperlinks key conclusions to their underlying source material.

Turbo Law

Those capabilities should be tested on the buyer's work.