Incident Report: Provider Timeout, 14 November 2025

This is the customer incident report for 14 November 2025, with internal detail we can publish. Customer names remain anonymized.

Summary

Between 06:14 and 06:25 UTC, Anthropic's Claude endpoints returned 503 and elevated timeout_ms beyond our hop budget for a subset of quality_optimized chains. Eridian failover promoted GPT-4o, then Gemini 2.0 Pro per policy. Platform availability stayed within SLO. Two legal templates that were pinned to Claude (no fallback, by customer policy) paused at the human gate instead of silently changing model.

Customer-facing severity: SEV-3 for unpinned traffic (elevated latency, some retries). SEV-2 for pinned legal templates that did not complete until Claude recovered or an approver unpinned.

Timeline (UTC)

• 06:14 Error rate on claude-sonnet-5 hops crosses 12% rolling 2-minute window. Page to Dubai on-call. Hong Kong overlap on desk.
• 06:15 Routing engine marks Claude degraded for unpinned policies. Fallback hops start.
• 06:17 Two tpl_msa_v3 runs in a European legal project stall: error.code model_pinned_unavailable. Dual control required to unpin.
• 06:19 Status component inference-anthropic set to degraded on geteridian.com/status.
• 06:25 Claude error rate returns below 2%. Degraded flag clears after 5-minute hold.
• 06:31 Pinned runs resume. One matter waited for counsel to reject unpin and wait; that was the correct control.
• 07:10 Post-incident export sent to affected tenants: request_id list, hop timings, cost delta from fallback models.

What worked

Fallback chains for unpinned traffic did what they are for. Cache hits were unaffected. Residency pins held; we did not open an off-region Claude hop to "save" a request.

Cost attribution showed a 19% unit-cost increase on the failed-over window because GPT-4o was the second hop. Finance teams got that number the same day, not at month end.

What we changed

• Pinned templates now show a louder in-product state: awaiting_model versus awaiting_approver. Counsel told us those were collapsing into one spinner.
• We added a runbook step: if pin hold exceeds 10 minutes, page the project's legal owner, not only SRE.
• Shadow evals for the fallback models on tpl_msa_v3 were already green. We still will not auto-unpin. Auto-unpin is a silent model change on a live matter.

What we will not do

We will not add a hidden fourth hop in another region to ride out a provider incident. We will not treat a 503 as permission to skip PII redaction or dual control. The outage was eleven minutes. The control failures we refuse would have lasted years in an exam pack.