Your legal AI test may already contain the answer

Start with the decision the team faced

Choose one task and write down what a successful answer needs to do before running the AI. For example, let’s say you’re testing whether it can find an earlier approved position for a recurring contract issue and explain when a lawyer needs to consider an exception. The practice partner defines the acceptable position, supporting evidence and conditions that should send the question back to a lawyer.

Use a short, approved task description containing only the facts needed to ask the question. Don’t put the eventual negotiated answer into the prompt. Keep the partner’s reference answer and scoring notes in a separate review location that the answering system can’t search. Otherwise, the reference meant to judge the answer becomes another way to supply it.

Record when the original task arose and choose the historical evidence cutoff. For the method used in our corporate engagement, the searchable reference set came from before the deal, and the test matter’s own documents were withheld. Apply both rules. An early draft from the same matter still fails the matter exclusion even if its date is old enough.

This defines a test of finding and applying earlier knowledge. A test that asks AI to review an uploaded draft has a different input boundary. If that’s your intended workflow, document the draft version available at the original task and keep later revisions and the final answer separate. Don’t mix those two designs under one score, because they give the system different information.

Check the contents, not just the folder date

In our example, Matter M-042 began on April 1, 2025, and the team needed an answer on May 6. The test searches only approved reference material available before April 1. An approved February position from an earlier matter is eligible if the test user can access it. M-042’s May 20 closing document is excluded because it belongs to the test matter and contains information learned later.

Now suppose someone copied that closing document into an older precedent folder. Filtering by the folder’s date would leave the answer available. The knowledge owner therefore needs to check each source’s actual version and origin, including copied passages, summaries and approved positions derived from the test deal. A new filename doesn’t make the content independent.

Keep an allowlist, which is the exact set of source versions the test may retrieve. For each source, save its ID, version, original matter, date it became available, and the reason it qualifies. A file hash, a fingerprint of its contents, helps show that the bytes didn’t change between runs. It doesn’t prove when the document existed, so keep the dated version history or other record that supports the cutoff too.

If the owner can’t establish a source’s origin or historical version, leave it out of this test until that gap is resolved. If the remaining material can’t support the expected answer, revise the task or mark it unanswerable from the allowed evidence before running it. That keeps an impossible question from being counted as a model failure.

The same review gives the sponsor a clearer investment decision. A strong answer earned from eligible evidence can support further testing. An answer copied from the completed deal can’t establish readiness for new work, so it shouldn’t be the reason to expand the tool. The wider investment choice still starts with the firm’s growth goal.

* Eight-week delivery, or twelve weeks when substantial data foundations are required, as agreed before work begins. If a covered stage fails its agreed benchmarks, we provide up to four additional weeks of remedy work at no charge, then waive or refund that stage’s fee if it still fails. Successfully delivered stages remain payable. Agreed client-caused delays pause the clock. Delivery guarantee terms.

Make the boundary hold through the whole run

Give the technical owner the task, source allowlist, matter exclusions and test user’s permissions. Use a separate test collection or enforce those rules before retrieved text reaches the model. A request in the prompt to ignore later documents leaves the documents available, which is why the boundary needs to be enforced by the system supplying the evidence.

Start a fresh session without earlier answers. Check every route that can add context: search indexes, uploaded files, connected tools, saved conversation history and cached answers. If the test claims to use only the approved reference set, disable other retrieval routes for that run. An excluded closing summary in a saved conversation would supply the answer even if document search was configured correctly.

Run under the intended user’s access rights. Historical eligibility and permission are separate checks: an earlier source can still be restricted to another team. In our engagement, the firm’s security team connected its existing authorization system so access was checked before retrieval. Your test should also include a permitted test fixture that the test account isn’t allowed to read, so the owner can inspect whether the boundary holds without widening anyone’s real access.

Save the source IDs, versions and passages actually sent to the model, together with the prompt, model identifier, settings, retrieval configuration and output. Keep sensitive records in the firm’s approved environment with access limited to the reviewers who need them. A list of final citations alone is incomplete because the model may have received material it didn’t cite.

Inspect the retrieved evidence before grading the legal answer. If excluded material reached the model, mark the run invalid, repair the source boundary and rerun it. A polished answer doesn’t rescue a test whose input already contained the result.

Follow the leaked document through the repair

Here’s what that looks like in practice. The first M-042 run returns the final negotiated position and cites a document stored in the precedent folder. The source trace shows that the document is a copy of the May 20 closing record. The partner may agree with the answer, but the run can’t show that the system reached it from earlier knowledge.

The knowledge owner removes the copied record and checks for its summaries and extracts. The technical owner rebuilds the approved test collection, clears the earlier session and cached result, and saves the revised configuration. Those steps remove the known route by which the final answer entered this test.

The corrected answer may be less complete than the finished deal file because the original task was less complete too. In this example, asking for the missing fact is the expected behavior. Scoring only against the final wording would penalize the system for respecting the evidence it was given.

Have the partner assess whether the answer uses the right position, supports its reasoning and identifies the exception or missing information. Software can check that a source exists, a reference resolves and a quotation matches. The lawyer still judges whether that evidence supports the conclusion. Record answer quality, access failures and invalid runs separately so a single average doesn’t conceal a broken boundary.

Save a test record someone else can repeat

Use this manifest, a record of the test’s inputs and decision, for each task. The middle column carries the example through to review. Copy the final column for your own record.

One repaired question is a diagnostic result. Before approving a workflow, choose representative tasks and agree the acceptance rules with the practice partner, including supported answers, missing-evidence responses and restricted-source attempts. Keep some matters out of prompt development and tuning, then use them for the final check. Otherwise, repeated repairs can teach the team how to pass the examples it already knows.

Repeat the required runs and keep every result, including variations and failures. Record any known use of the test material in model training or earlier tuning. Filtering retrieval can’t remove an answer already learned by a model. Where that history is unknown, record the limit and strengthen the evidence with fresh matters the team hasn’t used for development. Don’t describe the test as free of every possible source of leakage.

Harvey’s Legal Agent Benchmark also frames evaluation around legal assignments, matter materials and work products assessed against expert criteria. Your firm’s manifest adds the local question those materials alone don’t answer: which exact evidence was this system allowed to see for this historical task?

Approve one clean test before funding expansion

Start with one closed matter and have the knowledge owner and practice partner sign off its completed manifest. That gives the technical owner a defined test to run and gives the sponsor a visible reason to accept, repair or hold the next stage. If the current platform can isolate sources and expose the necessary trace, use that capability. If it can’t, ask the provider to demonstrate the missing control before committing to a separate build.

After the representative test set passes, keep the first rollout within the accepted task and review scope. Measure the drafting and checking effort the workflow creates, including partner corrections. Rerun the relevant evaluation when the sources, permissions, prompt or model change, because each can change what the system sees or how it answers. If the source boundary fails, suspend the affected AI workflow and use the approved human review route while the owner repairs it.

A clean historical test gives your firm a sounder basis for choosing what to expand. It helps you avoid funding a convincing demonstration that leaves lawyers doing the same checking later. If your larger goal is growth with less repeated partner work, the next decision is where that work most limits the firm and which change is worth implementing.

* Eight-week delivery, or twelve weeks when substantial data foundations are required, as agreed before work begins. If a covered stage fails its agreed benchmarks, we provide up to four additional weeks of remedy work at no charge, then waive or refund that stage’s fee if it still fails. Successfully delivered stages remain payable. Agreed client-caused delays pause the clock. Delivery guarantee terms.