Multi-Model Routing: How to Build a Fault-Tolerant Inference Stack

Model selection shouldn't be a one-time architectural decision made in a Q3 planning meeting. Quality, cost, and latency profiles shift with every provider release. Eridian's routing engine evaluates each request against your policy: cost_optimized, latency_optimized, quality_optimized, or a custom policy you define.

Fallback Chains

Every routing policy includes an ordered fallback chain. If Claude Sonnet times out or returns a 503, traffic fails over to GPT-4o, then Gemini Pro - automatically, within the same request. Your application never sees provider-level failures unless every model in the chain is down.

{
  "routing_policy": "quality_optimized",
  "fallback_chain": ["claude-sonnet-5", "gpt-4o", "gemini-2.0-pro"],
  "timeout_ms": 30000
}

Routing decisions are logged in the observability dashboard with full attribution: which model handled the request, whether fallback was triggered, latency per hop, and cost comparison against the primary model.

Routing is a control problem disguised as a model selection problem. You are balancing cost ceilings, latency SLOs, quality floors, provider availability, and regulatory pinning rules simultaneously on every request.

Quality_optimized does not always mean largest model. The engine scores candidate models using rolling eval metrics weighted by workflow type.

Fallback chains are ordered lists, not sets. Operations teams document chain rationale the same way they document DNS failover - primary, secondary, tertiary, with explicit timeouts per hop.

Budget-aware routing deprioritizes expensive models when project spend crosses soft thresholds even if quality_optimized would otherwise select them.

Pinning is common for legal workflows under model approval processes. Pinning still benefits from caching and observability; it only disables automatic model substitution.

Latency SLO breaches trigger automatic promotion of faster models for the offending workflow class when adaptive routing is enabled on Enterprise tier.

Provider rate limits surface as structured errors with retry-after hints. The routing engine respects Retry-After headers before attempting the next fallback candidate.

Shadow routing sends duplicate requests to candidate models asynchronously for offline quality comparison without affecting production responses.

Telemetry fields eridian.route_fallback and eridian.routing_policy should appear in every executive dashboard alongside spend.

Adaptive routing learns from weekly offline eval jobs submitted by customer ML teams; human approval required before production promotion.

Cost_optimized policies still respect quality_floor; violating floor triggers fallback rather than silently serving low-quality outputs.

Cross-region routing respects data residency tags on projects; US-only tenants never route to EU inference endpoints.

Provider maintenance windows ingest status feeds; models in maintenance are removed from candidate pools proactively.

Routing explainability API returns top three candidates with scores for support investigations.

Routing policies are code-adjacent configuration: store them in version control, review them in PRs, and deploy them with the same discipline as application changes.

Routing is a control problem disguised as a model selection problem. You are balancing cost ceilings, latency SLOs, quality floors, provider availability, and regulatory pinning rules simultaneously on every request.

Quality_optimized does not always mean largest model. The engine scores candidate models using rolling eval metrics weighted by workflow type.

Fallback chains are ordered lists, not sets. Operations teams document chain rationale the same way they document DNS failover - primary, secondary, tertiary, with explicit timeouts per hop.

Budget-aware routing deprioritizes expensive models when project spend crosses soft thresholds even if quality_optimized would otherwise select them.

Pinning is common for legal workflows under model approval processes. Pinning still benefits from caching and observability; it only disables automatic model substitution.

Latency SLO breaches trigger automatic promotion of faster models for the offending workflow class when adaptive routing is enabled on Enterprise tier.

Provider rate limits surface as structured errors with retry-after hints. The routing engine respects Retry-After headers before attempting the next fallback candidate.

Shadow routing sends duplicate requests to candidate models asynchronously for offline quality comparison without affecting production responses.

Telemetry fields eridian.route_fallback and eridian.routing_policy should appear in every executive dashboard alongside spend.

Adaptive routing learns from weekly offline eval jobs submitted by customer ML teams; human approval required before production promotion.

Cost_optimized policies still respect quality_floor; violating floor triggers fallback rather than silently serving low-quality outputs.

Cross-region routing respects data residency tags on projects; US-only tenants never route to EU inference endpoints.

Provider maintenance windows ingest status feeds; models in maintenance are removed from candidate pools proactively.

Routing explainability API returns top three candidates with scores for support investigations.

Routing policies are code-adjacent configuration: store them in version control, review them in PRs, and deploy them with the same discipline as application changes.

Review fallback chains after every provider pricing change; economic ordering shifts silently otherwise.

Publish routing policy diffs to internal change advisory boards when quality floors move.

Track fallback rate per model; chronic secondary routing indicates primary model instability.

Validate residency tags on projects during onboarding before enabling auto routing.

Benchmark adaptive routing proposals against fixed policies for four weeks before promotion.

Include routing explainability screenshots in customer QBR decks when spend variance exceeds ten percent.

Disable experimental models in production policies until eval harnesses cover regression suites.

Coordinate routing maintenance windows with customer success for pinned-model tenants.

Log user-visible model names only when policy allows; some tenants require opaque routing labels.

Rehearse manual policy rollback steps quarterly; routing misconfiguration is a high-impact change class.

Publish model candidate scorecards after major provider launches before enabling auto selection.

Track cost delta between primary and fallback models monthly for FinOps reconciliation.

Validate timeout budgets against p99 provider latency with safety margin for chained fallbacks.

Document pinned-model exceptions with expiry dates to prevent eternal bypass of routing engine.

Run shadow evals when legal approves new model families for restricted workflow classes.

Include routing policy identifiers in change logs sent to customer security contacts.

Test rate-limit interaction with routing failover under burst traffic in load harnesses.

Align routing dashboards with status page component boundaries for faster incident comms.

Review adaptive routing promotions in CAB meetings with rollback triggers predefined.

FinOps tags routes with internal cost center codes via metadata for showback/chargeback reports.

Engineering managers compare quality eval scores across models quarterly before policy changes.

On-call runbooks list provider status pages and Eridian status components side by side.

Tenant admins receive alerts when fallback rate exceeds baseline for twelve consecutive hours.

Architecture review boards approve new model families before they enter production routing pools.

Routing engineers document quality_floor calibration datasets with version control tags.

Provider account teams receive monthly failover summaries for relationship management.

Latency regressions trigger automatic comparisons against previous routing policy version.

Customers pin critical workflows during provider conferences when traffic patterns are unpredictable.

Routing configs lint in CI for unreachable models or disallowed residency combinations.

Support macros link to explainability API examples for top three routing tickets weekly.

FinOps dashboards show counterfactual spend if primary model handled all requests without fallback.

Routing on-call rotations pair with provider liaison contacts in escalation matrices.

New hires trace a single request through routing logs as part of onboarding week one.

Routing policies include annotated comments explaining weight choices for future maintainers.

Customers receive diff emails when default enterprise policy templates update.

Load tests validate routing under burst concurrent fallbacks without retry storms.

Routing metrics feed capacity models forecasting provider commit consumption.

Governance boards review adaptive routing promotions alongside model risk assessments.

Routing simulation environments replay production traffic samples with identifiers scrubbed to validate policy drafts before promotion.

Multi-region tenants configure routing graphs that prefer in-region models unless quality_floor would be violated by local pool exhaustion.

Routing policies include explicit deny lists for models pending legal approval even if providers announce general availability.

Executive dashboards show percentage of spend routed to each provider family week over week with anomaly highlighting.

Routing configuration reviews occur before major marketing events when traffic spikes could exhaust cheaper model pools first.

Customers maintain separate routing policies for batch and interactive traffic to optimize cost and latency independently without compromise.

Provider relationship managers receive monthly reports of fallback frequency attributed to upstream incidents versus policy constraints for accountability.

Routing tests include injected latency faults to validate timeout propagation without retry storms across fallback chains.

Model pinning documentation requires named business owner, approval date, and scheduled review when provider contracts renew.

FinOps compares realized routing savings against counterfactual single-model baseline using exported eridian cost attribution tables.

Routing on-call training includes hands-on labs promoting and demoting models in staging policies with measurable before-and-after latency deltas.

Customers receive quarterly routing health reports highlighting chronic fallback patterns suggesting provider or policy misalignment.

Routing engine logs include machine-readable reason codes consumed by automated remediation playbooks for known failure classes.

Tenant-specific routing dashboards highlight when pinned models diverge from platform recommended pools for renewal conversations.

Routing policy simulators estimate monthly spend impact before changes affecting more than ten percent of traffic volume.

Global routing incidents trigger status component updates on the public status page with customer-facing narrative within thirty minutes of detection.

Routing engineers pair with FinOps during provider contract renegotiations to align commit discounts with observed route distribution trends.

Customers testing new models in staging use shadow routing policies until quality evals meet production promotion gates.

Routing change tickets require linked observability dashboard screenshots demonstrating baseline and expected delta.

Routing SLO reviews occur monthly with provider liaison attendance mandatory.