Model Evals That a Committee Can Read

Model risk committees do not sign BLEU scores. They sign whether a workflow is allowed to run on live matters. If your eval report requires an engineer in the room to translate it, you do not have an eval program. You have a lab notebook.

We rebuilt our eval scorecards so the same artifact can go to engineering, legal, and the committee.

What a scorecard contains

For each production workflow template:

• Task: the job in one sentence (extract MSA termination, summarize incident, map clause to PDPL article)
• Gold set: labeled items with owners and review date
• Models in the routing chain, with the pinned version IDs
• Metrics the committee asked for, not the ones that look good in a launch post
• Failure examples, including the ones we did not cherry-pick
• Decision: allow, allow with human gate, pin to model, or block

{
  "eval_id": "eval_legal_msa_term_2025q2",
  "template": "tpl_msa_v3",
  "n": 400,
  "span_f1": 0.931,
  "policy_mismatch": 0.012,
  "human_gate_required": true,
  "decision": "allow_with_gate"
}

policy_mismatch is the rate at which the model cites a playbook section that the gold set says is wrong. That number matters more than fluency.

Gold sets are operational assets

A gold set that never updates is a vanity metric. Legal playbooks change. We require a named owner and a refresh interval. If the gold set is stale past the interval, the scorecard status is expired, and new model pins are blocked until it is refreshed.

Refreshing is dual control when the set is used for a production allow decision. Engineers can propose diffs. Counsel accepts them.

Shadow evals before a route change

When a provider ships a new model, we do not flip the chain because latency improved. We run the new candidate in shadow against the gold set and against two weeks of production traces (redacted). Only then does a pin change go to dual control.

In June 2025 a candidate improved span F1 by 0.8 points and doubled policy_mismatch on indemnity language. We did not promote it. The latency graphs were prettier. The committee would not have signed them.

What we will not put on the scorecard

We will not report "win rate versus GPT" as a governance metric. Institutions are not buying a leaderboard. We will not hide a regression in a blended average across unrelated tasks. Each production template has its own card.

If a workflow cannot produce a scorecard a non-engineer can read in ten minutes, it stays in sandbox.