Skip to main content

Deterministic Exact-Match Cache

GoRoute AI Gateway embeds a tenant-isolated, deterministic exact-match response cache that serves identical queries in 0.05 milliseconds with zero upstream token spend.

No (stochastic) Yes (deterministic) HIT MISS Incoming Chat Request temperature == 0.0? Direct Upstream Inference Compute SHA-256 Digest:SHA256(OrgID + Model + System + Messages + Tools) Cache HIT in Redis / Memory? Return Cached Response(0.05ms, $0.00 Cost, X-Gateway-Cache: HIT) Store Response in Cache (TTL: 1h)

1. Deterministic Hashing Key​

To guarantee tenant isolation and output correctness, the cache key is constructed using a cryptographic SHA-256 digest over all canonical output-affecting request fields (organization, model, messages, system prompt, tools, sampling/output parameters, and extra_body):

CacheKey = SHA256(OrgID || Epoch || CanonicalRequestFields)
  • Tenant Isolation: An identical prompt submitted by Org A will never be returned to Org B.
  • Cross-Replica Invalidation Epochs: Calling POST /admin/v1/cache/invalidate increments a Redis-backed organization invalidation epoch ({org:<org_id>} hash-tagged) so cached entries are invalidated immediately across all gateway replicas.
  • Deterministic Only: Requests with temperature > 0.0 or non-zero randomness bypass the cache to preserve LLM creativity and variance.

2. Benchmark Comparison​

MetricUpstream Model CallGoRoute Cached Hit
Response Latency450 ms – 1,800 ms0.05 ms (50 µs)
Cost IncurredFull token billing ($0.001 – $0.05)$0.00 (Zero tokens)
Upstream Network I/O100%0% (Bypassed)
Telemetry HeaderX-Gateway-Cache: MISSX-Gateway-Cache: HIT