Skip to main content

AI Gateway: Traffic Management & Model Routing

The gateway is the traffic director between your application and its model providers. Your application makes requests through a consistent interface, and GoRoute handles provider selection, retries, and failover behind it. Experienced operators can also tune routing, caching, and operation-level compatibility.

AI request GoRoute router Selected provider Fallback provider

GoRoute currently provides 73 ProviderType profiles (8 native adapters, 60+ built-in and extended OpenAI-compatible profiles, and CustomOpenAICompatible) through its unified API.


Core Capabilities​

CapabilityImplementationGuide
Unified Data Plane (24 /v1 Paths)OpenAI-compatible /v1/* & /openai/v1/*, plus native Anthropic /anthropic/v1/messages & count_tokens, Responses API, Realtime WebSockets, POST /v1/guardrails/apply, Audio, Images, Files, Batches, Rerank, and Moderations.Chat Completions · Messages · Responses · Realtime
73+ Provider Profiles & SupportedOperationsNative adapters for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Gemini, Google Vertex, Cohere, Mistral, Ollama, plus 60+ OpenAI-compatible profiles with SupportedOperations enforcement (OperationSupportChecker).Providers Overview · OpenAI-Compatible
Stage 1 Complexity ClassifierHeuristic prompt complexity scoring routes simple requests to fast/low-cost tiers and complex tasks to reasoning models automatically.Routing & Load Balancing
Stage 2 Load BalancersSix strategies across sibling deployments: weighted-shuffle, least-busy, latency-based (EWMA), usage-based-headroom, lowest-cost, and round-robin.Routing & Load Balancing
3-State Redis Circuit BreakersShared Closed → Open → Half-Open state in Redis across gateway replicas with atomic half-open probe leases.Resiliency & Fallbacks
Bounded Fallback ChainsConfigurable intra-group exponential retries and cross-model failover chains for 429/5xx/503 resilience.Resiliency & Fallbacks
2-Tier Exact-Match CacheL1 in-process memory + L2 Redis SHA-256 exact-match response caching for sub-millisecond cache hits.Exact-Match Caching

Request Routing Pipeline​

Miss 429 / 5xx Hit Client / SDK 2-Tier Exact-Match Cache(L1 Memory + L2 Redis) Stage 1: Complexity Classifier(fast vs. reasoning alias) Stage 2: Load Balancer(weighted-shuffle / least-busy / latency-based /headroom / lowest-cost / round-robin) 3-State Redis Circuit Breaker& SupportedOperations Check 73+ Upstream Providers Bounded Retries &Fallback Chain

Explore Data Plane & Routing Guides​