Semantic Caching: The Untapped 40% Cost Reduction in LLM Inference

Every enterprise AI deployment eventually hits the same wall: inference costs scale linearly with usage, but query patterns are far from random. Legal teams ask similar contract questions. SREs run similar incident summaries. Developers re-run nearly identical code review prompts. Exact-match caching misses most of this duplication. Semantic caching doesn't.

How Semantic Caching Works at Eridian

When a request arrives at the Eridian gateway, we embed the prompt using a lightweight embedding model - not the frontier model you're paying for. That embedding is searched against a project-scoped vector store of previously cached prompt-response pairs. If cosine similarity exceeds your configured threshold (default: 0.92), we return the cached response immediately.

{
  "cache_config": {
    "enabled": true,
    "similarity_threshold": 0.92,
    "ttl_seconds": 3600,
    "scope": "project",
    "exclude_features": ["structured_output"]
  }
}

Measuring the 38% Reduction

We analyzed 90 days of production traffic from a $300B global bank running Eridian Dev across 1,200 engineers. Before semantic caching, their monthly inference bill was $312K. After tuning the similarity threshold to 0.91 and scoping cache to project-level (not global), hit rate stabilized at 34.7% and costs dropped to $193K - a 38% reduction with no measurable quality regression on their internal eval suite.

The cache isn't free - you pay for embeddings and vector storage. But embedding cost is roughly 1/100th of frontier model cost. The math works.

Tuning the Threshold

• 0.85-0.90: Aggressive caching. More hits, but near-miss responses may diverge slightly from a fresh inference.
• 0.90-0.95: Balanced. Our default. Strong hit rate with high accuracy on financial services evals.
• 0.95-0.98: Conservative. Fewer hits, but virtually identical responses.

What We Exclude from Cache

Structured output with dynamic JSON schemas, requests with RAG context that changes per document upload, and any request where pii_config differs from the cached entry are excluded by default. Cache invalidation is event-driven - when you update a prompt template version, related cache entries are purged automatically.

Semantic caching is not a silver bullet. It won't help one-off creative tasks. But for enterprise workflows with repetitive patterns - contract review, incident triage, code review - it's the highest-ROI optimization we've shipped.

Enterprise RAG workloads repeat structural patterns: summarize this clause, extract obligations from this MSA, compare indemnification language across vendors. Exact-match caches fail because whitespace, ordering, and minor phrasing changes produce different hash inputs while semantic intent stays identical.

Eridian embeds prompts with a dedicated embedding model sized for cost efficiency - not the frontier model your application selected for quality. Embedding latency typically adds 15-40ms, which is negligible compared to a 1.2s model round trip but decisive for cache lookup economics.

Vector indexes are scoped per project by default. Global scope shares hits across projects inside an organization when prompts are intentionally standardized - for example, firm-wide compliance checklists used by both legal and ops teams.

Cache entries store the redacted prompt fingerprint, model route, template version, and PII configuration hash. If any of those differ, lookup skips the entry even when embedding similarity is high.

Operations teams monitor cache hit rate alongside p95 latency. A sudden drop usually indicates index rebuild, template version churn, or a new workflow with genuinely novel prompts - not a model outage.

Financial services customers often ask whether cached responses can leak data across tenants. Tenant isolation is enforced at the storage partition level: embeddings and payloads never cross organization boundaries.

Tuning similarity thresholds should be empirical, not ideological. Start at 0.92, run shadow comparisons for two weeks, and measure answer equivalence on a labeled eval set.

Semantic caching composes with prompt compression and routing optimization. Combined, these optimizations produced $35.4K monthly savings for a representative enterprise tenant in our June 2025 attribution report.

Invalidation is event-driven: template publishes, RAG corpus updates, and manual purges by project admins. TTL defaults to one hour for interactive workflows and twenty-four hours for batch reporting jobs.

When designing SLAs, treat cache hits as a distinct latency class. Hits often complete in under 20ms end-to-end, which changes capacity planning for synchronous user interfaces.

Cache key material includes template_version so prompt refactors do not silently serve stale policy language to compliance reviewers.

Hybrid workloads mixing RAG and non-RAG prompts should include document_version in the fingerprint; otherwise two users querying different corpora could share answers.

Eridian exposes cache_similarity on misses to help developers understand near-misses during threshold tuning.

For batch ETL-style inference, consider longer TTL with explicit invalidation on upstream dataset version bumps.

Embedder model upgrades trigger background re-embedding jobs; hit rate may dip while indexes warm after deploys.

FinOps teams correlate cache hits with monthly invoices by exporting eridian.cache_hit and cost_usd fields into their warehouse. Spreadsheets that only track total tokens miss the story.

Developer education matters: teams that treat the cache as magic often disable it after one bad near-miss. Runbooks should explain similarity scores and template versioning before toggling features off.

Cold-start after deploys can be mitigated by pre-warming common templates from staging traffic captures. Eridian supports export/import of cache statistics for capacity planning.

Legal reviewers sometimes require proof that cached answers reference the same policy version as live templates. Audit logs include template_version on every hit and miss.

SRE dashboards page when hit rate drops below rolling seven-day baseline by more than ten points during business hours.

Hybrid search RAG prompts should never share cache entries across different document sets even when questions match verbatim.

Batch reporting jobs benefit from extended TTL but require explicit invalidation hooks when upstream numbers change materially.

Embedding quantization reduces index size with minimal recall impact; operations teams validate recall quarterly on labeled sets.

Semantic caching is disabled automatically when structured_output schema changes mid-request retry loop to prevent schema mismatch responses.

Enterprise RAG workloads repeat structural patterns: summarize this clause, extract obligations from this MSA, compare indemnification language across vendors. Exact-match caches fail because whitespace, ordering, and minor phrasing changes produce different hash inputs while semantic intent stays identical.

Eridian embeds prompts with a dedicated embedding model sized for cost efficiency - not the frontier model your application selected for quality. Embedding latency typically adds 15-40ms, which is negligible compared to a 1.2s model round trip but decisive for cache lookup economics.

Vector indexes are scoped per project by default. Global scope shares hits across projects inside an organization when prompts are intentionally standardized - for example, firm-wide compliance checklists used by both legal and ops teams.

Cache entries store the redacted prompt fingerprint, model route, template version, and PII configuration hash. If any of those differ, lookup skips the entry even when embedding similarity is high.

Operations teams monitor cache hit rate alongside p95 latency. A sudden drop usually indicates index rebuild, template version churn, or a new workflow with genuinely novel prompts - not a model outage.

Financial services customers often ask whether cached responses can leak data across tenants. Tenant isolation is enforced at the storage partition level: embeddings and payloads never cross organization boundaries.

Tuning similarity thresholds should be empirical, not ideological. Start at 0.92, run shadow comparisons for two weeks, and measure answer equivalence on a labeled eval set.

Semantic caching composes with prompt compression and routing optimization. Combined, these optimizations produced $35.4K monthly savings for a representative enterprise tenant in our June 2025 attribution report.

Invalidation is event-driven: template publishes, RAG corpus updates, and manual purges by project admins. TTL defaults to one hour for interactive workflows and twenty-four hours for batch reporting jobs.

When designing SLAs, treat cache hits as a distinct latency class. Hits often complete in under 20ms end-to-end, which changes capacity planning for synchronous user interfaces.

Cache key material includes template_version so prompt refactors do not silently serve stale policy language to compliance reviewers.

Hybrid workloads mixing RAG and non-RAG prompts should include document_version in the fingerprint; otherwise two users querying different corpora could share answers.

Eridian exposes cache_similarity on misses to help developers understand near-misses during threshold tuning.

For batch ETL-style inference, consider longer TTL with explicit invalidation on upstream dataset version bumps.

Embedder model upgrades trigger background re-embedding jobs; hit rate may dip while indexes warm after deploys.

FinOps teams correlate cache hits with monthly invoices by exporting eridian.cache_hit and cost_usd fields into their warehouse. Spreadsheets that only track total tokens miss the story.

Developer education matters: teams that treat the cache as magic often disable it after one bad near-miss. Runbooks should explain similarity scores and template versioning before toggling features off.

Cold-start after deploys can be mitigated by pre-warming common templates from staging traffic captures. Eridian supports export/import of cache statistics for capacity planning.

Legal reviewers sometimes require proof that cached answers reference the same policy version as live templates. Audit logs include template_version on every hit and miss.

SRE dashboards page when hit rate drops below rolling seven-day baseline by more than ten points during business hours.

Hybrid search RAG prompts should never share cache entries across different document sets even when questions match verbatim.

Batch reporting jobs benefit from extended TTL but require explicit invalidation hooks when upstream numbers change materially.

Embedding quantization reduces index size with minimal recall impact; operations teams validate recall quarterly on labeled sets.

Semantic caching is disabled automatically when structured_output schema changes mid-request retry loop to prevent schema mismatch responses.

Instrument similarity tuning workshops with legal, ops, and engineering stakeholders so threshold changes receive cross-functional sign-off.