Evaluations

Evaluations measure prompt templates, routing policies, and model pins against a labeled set before those objects serve production traffic. Runs are versioned, attributed to an actor, and exportable as a PDF or JSON pack a model-risk committee can read without opening a notebook.

Create a Suite

POST /v1/evals/suites
Authorization: Bearer eridian_sk_...
Content-Type: application/json

{
  "project_id": "prj_legal_001",
  "name": "msa-termination-v4",
  "template_id": "tpl_contract_summary_v3",
  "dataset_id": "ds_msa_gold_12",
  "metrics": ["exact_match", "rubric_legal", "pii_leak"],
  "models": ["gpt", "gemini", "auto"]
}

Required scope: evals:write

Dataset

Upload labeled cases as JSONL. Each line is one case:

{
  "id": "case_001",
  "input": { "document": "...", "focus_area": "termination" },
  "expected": { "severity": "high" },
  "rubric": "Must flag missing cap on indemnification."
}

POST /v1/evals/datasets
Authorization: Bearer eridian_sk_...
Content-Type: multipart/form-data

Datasets are residency-pinned with the project. They never train GPT or Gemini. They are used only for scored inference through the same gateway as production.

Run

POST /v1/evals/suites/evs_msa_term_v4/runs
Authorization: Bearer eridian_sk_...

{
  "id": "evr_19c4",
  "object": "eridian.eval.run",
  "status": "running",
  "models": ["gpt", "gemini", "auto"],
  "case_count": 120
}

Each case is an ordinary inference call: PII redaction, residency, and routing policy apply. metadata.eval_run_id is set so usage can be excluded from production FinOps or tagged as eval spend.

Results

GET /v1/evals/runs/evr_19c4
Authorization: Bearer eridian_sk_...

{
  "id": "evr_19c4",
  "object": "eridian.eval.run",
  "status": "completed",
  "scores": {
    "gpt": { "exact_match": 0.81, "pii_leak": 0.0 },
    "gemini": { "exact_match": 0.79, "pii_leak": 0.0 },
    "auto": { "exact_match": 0.84, "pii_leak": 0.0 }
  }
}

Committee Report

POST /v1/evals/runs/evr_19c4/report
Authorization: Bearer eridian_sk_...

Returns object: eridian.eval.report with pass/fail against thresholds, sample failures, routing policy ID, template version, residency region, and the actor who launched the run. Attach the report ID to a dual-control request when promoting a template from draft to published.

Thresholds

{
  "promote_if": {
    "exact_match": { "min": 0.8 },
    "pii_leak": { "max": 0.0 }
  }
}

Failed thresholds block template publish when evals_required: true is set on the project.

Python

run = client.evals.runs.create(
    suite_id="evs_msa_term_v4",
    models=["gpt", "gemini", "auto"],
)
print(run.id, run.status)

See Prompt Templates for versioning and Dual Control for promotion gates.