ARCHITECTURE

Why Your LLM Inference Pipeline Is Broken (And How to Fix It)

James Okafor · February 28, 2025 · 15 min read

Most teams treat LLM inference as a thin wrapper around an OpenAI or Anthropic API call. Prompt in, response out. Ship it. This works until you have 14 teams, 6 model providers, no cost attribution, no audit trail, and a CISO asking why client data is leaving your VPC.

The Pipeline Is the Product

Eridian's architecture treats inference as a pipeline: schema normalization → routing → semantic cache → RAG → PII redaction → model inference → structured output parsing → observability. The model is an implementation detail. The pipeline is what your compliance team audits.

response = client.inference(
    model="auto",
    messages=[{"role": "user", "content": prompt}],
    project_id="prj_legal_001",
    features=["pii_redaction", "semantic_cache", "structured_output"],
)

print(response.eridian.route)       # claude-sonnet-5
print(response.eridian.cache_hit)   # True
print(response.eridian.pii_redacted) # 2
print(response.eridian.cost_usd)    # 0.0042

Three Failure Modes We See Repeatedly

No observability: Teams can't answer 'which team spent $40K last Tuesday?'
No governance: Every team picks their own model, their own API key, their own data handling.
No resilience: Single-model dependency means one provider outage takes down every workflow.

Fixing this doesn't require rebuilding your applications. Eridian is OpenAI-compatible and Anthropic-compatible. Point your SDK at our gateway. The pipeline runs transparently. Your app code changes minimally; your infrastructure changes completely.

Calling a model API directly gives you tokens. Calling Eridian gives you an attributable, governable unit of work. That distinction matters when legal asks for provenance and finance asks for cost allocation by team, feature, and workflow.

Schema normalization means your client can speak OpenAI-style messages while the routing layer translates to provider-native formats. Your integration code stays stable when providers revise parameter names or response envelopes.

Pipeline stages emit spans compatible with OpenTelemetry. Each span includes project_id, key_id, template version, and feature flags active for the request.

Failure isolation is per stage. A RAG retrieval timeout does not silently disable PII redaction; the request fails explicitly with error.code rag_timeout unless degraded_mode is configured - off by default for regulated tenants.

Batch and streaming paths share governance hooks. Streaming still records final token counts, cost, and route selection when the client closes the SSE connection.

Migration from direct OpenAI or Anthropic usage typically requires changing base URL and API key, then enabling features incrementally: observability first, caching second, PII third, routing fourth.

Multi-step workflows should pass consistent metadata across calls so audit exports correlate sub-steps. Use workflow_run_id in metadata for agentic graphs.

Capacity planning must include embedding throughput when semantic caching is enabled. Eridian scales embedding workers independently from model gateway workers.

Gateway workers authenticate keys before any pipeline stage executes, preventing unauthenticated traffic from touching embedding or retrieval subsystems.

Structured output retries append corrective system messages; each retry is billed and logged separately for cost transparency.

Webhook emissions fire after pipeline completion, not after model response alone, so downstream systems see final eridian metadata.

Rate limits apply at key and project granularity simultaneously; breaching either returns 429.

Shadow traffic for migration validation duplicates requests to Eridian while production still hits legacy providers until cutover checklist completes.

Disaster recovery scenarios assume provider outage plus cache unavailability. Fallback chains and queue shedding policies are tested quarterly for enterprise contracts.

The pipeline is the product. Models are replaceable components inside it.