Deterministic Exact-Match Cache
GoRoute AI Gateway embeds a tenant-isolated, deterministic exact-match response cache that serves identical queries in 0.05 milliseconds with zero upstream token spend.
1. Deterministic Hashing Key
To guarantee tenant isolation and output correctness, the cache key is constructed using a cryptographic SHA-256 digest over all canonical output-affecting request fields (organization, model, messages, system prompt, tools, sampling/output parameters, and extra_body):
CacheKey = SHA256(OrgID || Epoch || CanonicalRequestFields)
- Tenant Isolation: An identical prompt submitted by Org A will never be returned to Org B.
- Cross-Replica Invalidation Epochs: Calling
POST /admin/v1/cache/invalidateincrements a Redis-backed organization invalidation epoch ({org:<org_id>}hash-tagged) so cached entries are invalidated immediately across all gateway replicas. - Deterministic Only: Requests with
temperature > 0.0or non-zero randomness bypass the cache to preserve LLM creativity and variance.
2. Benchmark Comparison
| Metric | Upstream Model Call | GoRoute Cached Hit |
|---|---|---|
| Response Latency | 450 ms – 1,800 ms | 0.05 ms (50 µs) |
| Cost Incurred | Full token billing ($0.001 – $0.05) | $0.00 (Zero tokens) |
| Upstream Network I/O | 100% | 0% (Bypassed) |
| Telemetry Header | X-Gateway-Cache: MISS | X-Gateway-Cache: HIT |