Semantic Caching

Semantic caching reduces redundant inference calls by matching prompts on meaning rather than exact string equality. Enterprise RAG and compliance workflows repeat structural patterns - summarize this clause, extract obligations, compare indemnification language - where minor phrasing changes produce different hash inputs while semantic intent stays identical.

How It Works

1. Embed: The incoming prompt is embedded using a dedicated lightweight embedding model (not the frontier model selected for quality).
2. Search: The embedding is searched against a vector store of previously cached responses scoped to your project or organization.
3. Match: If cosine similarity exceeds the threshold (default: 0.92), the cached response is returned without provider round trip.
4. Miss: If no match exceeds the threshold, the request routes to the model normally. The response is cached for future requests with fingerprint metadata.

Cache entries store the redacted prompt fingerprint, model route, template version, and PII configuration hash. If any of those differ, lookup skips the entry even when embedding similarity is high.

Configuration

{
  "cache_config": {
    "enabled": true,
    "similarity_threshold": 0.92,
    "ttl_seconds": 3600,
    "scope": "project",
    "exclude_features": ["structured_output"],
    "allow_structured_output": false
  }
}

Enable globally via project settings or per-request with "features": ["semantic_cache"].

Cache Headers

Response body field eridian.cache_hit mirrors the header. FinOps teams export cache_hit and cost_usd to attribute savings accurately.

Tuning the Threshold

• 0.85-0.90: Aggressive caching. More hits; near-miss responses may differ slightly. Run shadow comparisons before production.
• 0.90-0.95: Balanced default. Good hit rate with high accuracy for legal and ops templates.
• 0.95-0.98: Conservative. Fewer hits; use when answer equivalence must be near-identical.

Start at 0.92, run shadow comparisons for two weeks, and measure answer equivalence on a labeled eval set before tightening.

Invalidation

Invalidation is event-driven:

• Template publishes bump template_version in cache keys
• RAG corpus updates bump document version hashes
• Manual purges via DELETE /v1/cache?project_id=...
• TTL expiry for batch reporting jobs (often 24h vs 1h for interactive)

Scope: Global Cache

global scope shares hits across projects inside an organization when prompts are intentionally standardized - for example, firm-wide compliance checklists used by both legal and ops teams. Tenant isolation is enforced at the storage partition level; embeddings never cross organization boundaries.

Error Codes

See Inference for cache fields on responses and Usage for cost attribution of cache savings.