Runbooks

When GPT or Gemini sits on a production path, on-call owns the gateway, not the foundation model. These runbooks assume you have audit:read and dashboard access. Page the provider only after Eridian health says the family is failing independently of your project.

Severity

Contact for tenant escalations: contact@geteridian.com. Include X-Eridian-Request-Id values, project_id, and region.

SEV-1: Error rate

1. Open the observability dashboard for the project. Confirm whether error.code is model_unavailable, routing_timeout, rate_limit_exceeded, or budget_exceeded.
2. GET /v1/models/health. If gpt is unhealthy and gemini is healthy, set the project fallback chain to ["gemini", "gpt"] or pin critical templates to the healthy family.
3. If both families are healthy and you still 503, check residency: X-Eridian-Region versus the pin. A residency_violation storm means a client is sending the wrong region.
4. Do not disable PII redaction or fail_closed without dual control.
5. After mitigation, export the window via Evidence Export before logs roll.

Cache miss storm

1. Confirm X-Eridian-Cache-Hit drop on /v1/usage?group_by=feature.
2. Recent template publish invalidates cache keys. That is expected. Check template_version.
3. If a RAG reindex just completed, cache keys include document hashes; miss rate recovers as traffic repeats.
4. Do not lower similarity_threshold during an incident to "get hits back" without an eval.

Budget hard stop

1. Confirm 429 budget_exceeded versus rate_limit_exceeded.
2. If the desk must continue, a second approver raises the ceiling (Budgets, Dual Control).
3. If spend is a loop (client retry without idempotency), fix the client first. Raising the ceiling will not stop a retry storm.

Webhook backlog

1. GET /v1/webhooks/{id}/deliveries. If the consumer returns 500, Eridian will retry for 24 hours.
2. Disable the endpoint only if it is amplifying load. Replay after the consumer is healthy.
3. Verify signatures still match after a secret rotation. See Webhook Signing.

Provider timeout

If GPT or Gemini times out in-region:

• Application timeout must exceed the routing timeout_ms plus one fallback hop.
• eridian.route_fallback: true on successful responses is a signal, not an error.
• Open an incident with contact@geteridian.com if both families fail in the pinned region. Do not fail open to another region unless dual control approves a pin change.

Your services versus this gateway

These runbooks are for Eridian in the request path. Incident summarization, log analysis, and on-call assistance for your applications are Eridian Ops. Do not open a platform SEV-1 because payments-api 5xxed. Open an Ops incident. Page contact@geteridian.com only when GPT and Gemini are unhealthy in the pin or the gateway error rate is the customer-facing failure.

Generate an internal runbook (Ops vertical)

After you close a tenant incident in Eridian Ops, you can draft a service runbook from the timeline. That is not a substitute for this page.

POST /v1/ops/runbooks/generate
Authorization: Bearer eridian_sk_...

{
  "project_id": "prj_ops_001",
  "incident_id": "inc_2847",
  "model": "gpt",
  "title": "payments-api ledger timeout"
}

Promote those drafts with Evaluations and Dual Control if they auto-attach to a service. See Post-Mortems.

Follow-the-sun

Enterprise gateway support is staffed from Dubai, Hong Kong, and South Africa. Include region, project_id, and request ids. Do not send prompt bodies.

See Observability, Regions, On-Call, and Errors.