Eval corpus: cap raise, re-annotation, and three bugs it surfaced

2026-08-04. The corpus went from first-thirds of documents to mostly whole documents, the answer keys were rewritten against the new text, and the change exposed two real bugs plus one product defect. Read the last section before quoting any number from before this date.

What changed

The two changes had to ship together. Raising the cap alone gives the model two to three times more document to cover while an unchanged answer key still describes only the opening, so completeness falls for reasons that have nothing to do with review quality. Re-annotating alone was impossible, because the text the new risks describe was not in the corpus yet.

The excerpt marker

Every truncated entry now ends with an explicit [EXCERPT ENDS HERE...] marker. This is the load-bearing part of the fix, not the cap. The median fetched exhibit is about 44,000 characters and the largest is 687,000, so no affordable cap makes every document complete.

Before the marker, the pipeline reported the cut as a defect in the contract: "Document appears to be truncated", "No governing law, venue, injunctive relief, or remedies clauses visible". Six of the 18 risk findings across three NDAs were observations about the excerpt's condition rather than about contract risk, so a third of a short document's findings budget was being spent describing an artifact of our own fetcher, and the eval then scored the pipeline on what was left.

A second, separate defect fed the same problem: truncateAtBoundary took Math.max across a paragraph break, a sentence end and a bare line break. In hard-wrapped filing text the bare line break is nearly always the latest candidate, so it almost always won, and hard wraps land mid-sentence. The function did the opposite of its own docstring and left 22 of 25 excerpts ending mid-sentence. It now prefers paragraph, then sentence, then line.

What the new text contained

The re-annotation pass was not padding. Clauses that were entirely outside the old excerpts include:

• The whole of Potbelly's Section 13 standstill, including the don't-ask-don't-waive provision barring the receiving party from even requesting a waiver.
• Societal CDMO's February 2024 amendment making CoreRx and Merger Sub third-party beneficiaries entitled to enforce the agreement without having signed it.
• An intellectual property reservation in the ProFrac supply agreement that defines "Supplier Intellectual Property" using a sentence copied from the Purchaser paragraph, so the clause meant to reserve the supplier's rights reserves the purchaser's instead.
• An AppTech acceptance-test provision whose two halves set contradictory deadlines: accepted unless the customer objects within three days after the testing period, and accepted unless a rejection notice arrives before it ends.
• A forum clause fixing exclusive jurisdiction in "the State of Arizona, County of Phoenix", which is not a county.
• A Heartland surrender clause requiring the premises returned with ordinary wear and tear "accepted" rather than excepted, one wrong word that reverses the obligation.

Two corpus entries turned out to be the same agreement filed by both sides: the Flotek exhibit is the ProFrac supply agreement. All eleven ProFrac risks matched it verbatim, which is how it was confirmed rather than assumed. They remain as separate entries, but the vendor category is effectively carrying one document twice and should not be read as two independent samples.

Bug 1: the extraction timeout was sized for 2,000-word documents

ROLE_TIMEOUT_MS.extraction was 45s. A 5,045-word lease blew through it and failed over to the fallback provider four times in one contract, which is the silent quality downgrade this codebase treats as its worst failure mode.

Extraction is not a cheap-pass role. It runs key terms, obligations and completeness grading, and obligations alone returned 52 items on one 2,300-word lease. Latency tracks output size, not model tier.

This was failing in production, not only in the eval. Real uploads are routinely longer than the old corpus was, and nothing was measuring it. The only tell would have been the QUALITY DOWNGRADE console.error.

Raised to 120s, still well under the routes' 300s maxDuration. Verified on the worst case: the same 5,045-word lease now runs with zero failovers in 192.7s, 12 findings, 19 of 19 quotes verbatim.

Bug 2: the offline judge silently skipped most of the corpus

judge-saved-reviews.ts resolved a saved review to its answer key with SEC_CONTRACTS.find(c => `sec-${c.contractType}` === id). That matches only the first contract of each type. With eight contracts it skipped none; with the expanded corpus it skipped 25 of 33, and still exited 0 and printed a table.

A metric that quietly covers a quarter of what you think it covers is worse than no metric. It now derives ids exactly the way run-evals.ts does.

Defect 3: a review with a score and no findings

One review in nineteen returned risks.findings: [] while reporting overallRiskScore: 68, a 542-character summary naming three specific risks (uncapped CAM pass-throughs, rent acceleration on default, a 50 percent profit-share on assignment), 23 missing clauses, 8 recommendations and 19 obligations.

Every other pass produced content. Only the findings array came back empty, and the prose summary proves the model had identified the risks. A user would see a risk score of 68 with nothing behind it.

Not yet fixed, and not yet established as reproducible rather than transient. That is the next thing to check, and it matters more than any completeness number here: a scored review with no findings is broken output, not a weak one.

Method note: stopping a background eval does not stop it

Three eval processes ran concurrently without anyone intending it. TaskStop reported success on two of them and neither died; they kept calling the API and writing into the same output directory. One of them predated the cap change, so it was reviewing the old 14,000-character corpus while writing files that looked like current results.

It was caught by comparing a saved review's document.wordCount against the corpus: 2,191 against 4,549 for the same contract. Those 18 reviews were discarded.

Verify the process is gone, do not trust the stop. On Windows:

Get-CimInstance Win32_Process -Filter "Name='node.exe'" |
  Where-Object { $_.CommandLine -match "run-evals" }

Cost of that mistake: a dollar figure for our own API spend, omitted here. What it bought was nothing usable.

Results: 9 contracts, all eight types, on the new corpus

Not a full baseline. The credit balance would not cover 33 contracts at the new document length, so this is a subset chosen to cover every contract type. Run after the extraction timeout fix.

Zero failovers across all nine, including the 5,045-word lease that produced four before the timeout fix. Every contract measured the intended Opus pipeline.

Quote fidelity, re-checked offline with the fixed checker: 109 of 111 verified (108 strict, 1 verbatim across a page break), 0 invented clauses, and 2 transcription slips where the model corrupted a word while copying:

• "discontinue their employment with, one, the other Party" where the document reads "with, or engagement by, the other Party". 950 of 970 characters correct, then a three-word substitution that leaves the clause nonsensical.
• "any other available role or remedy" where the document reads "right or remedy".

Neither invents an obligation, but both break the verbatim promise that the clause quote is a span copied from the contract. 98.2 percent verbatim is the honest number for this run, and it is not the same claim as the "113 of 113" figure quoted from the old corpus.

The two NDAs are the interesting rows. Potbelly reads 47 semantic against 50 on the old corpus, and Societal 64 against 43, while both answer keys roughly doubled. Holding position against a materially harder key is an improvement, but the run-to-run noise on a single contract is 17 points, so neither number should be quoted as a movement on its own.

Reading the numbers

Scores from before 2026-08-04 are not comparable to scores after it. Both the documents and the answer keys changed, and they changed in opposite directions for the metric: documents got longer, which gives the model more to find, while answer keys got roughly twice as long, which raises the bar for completeness.

Expect completeness to read lower than the old baseline. That is the eval getting harder and more representative, not the pipeline getting worse. The old baseline was also only ever valid for 12 of 33 contracts, because credit exhaustion zeroed the rest.

Two structural cautions that survive this change:

1. A single contract moved 17 points of semantic completeness across runs on identical code. Any before/after smaller than that is noise.
2. Completeness is bounded by findings count. With answer keys now at 11 to 19 risks and the pipeline returning 8 to 12 findings, recall is capped well below 100 even with perfect precision. Compare across runs, not against 100.