AI Gateway: Traffic Management & Model Routing
The gateway is the traffic director between your application and its model providers. Your application makes requests through a consistent interface, and GoRoute handles provider selection, retries, and failover behind it. Experienced operators can also tune routing, caching, and operation-level compatibility.
GoRoute currently provides 73 ProviderType profiles (8 native adapters, 60+ built-in and extended OpenAI-compatible profiles, and CustomOpenAICompatible) through its unified API.
Core Capabilities
| Capability | Implementation | Guide |
|---|---|---|
Unified Data Plane (24 /v1 Paths) | OpenAI-compatible /v1/* & /openai/v1/*, plus native Anthropic /anthropic/v1/messages & count_tokens, Responses API, Realtime WebSockets, POST /v1/guardrails/apply, Audio, Images, Files, Batches, Rerank, and Moderations. | Chat Completions · Messages · Responses · Realtime |
73+ Provider Profiles & SupportedOperations | Native adapters for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Gemini, Google Vertex, Cohere, Mistral, Ollama, plus 60+ OpenAI-compatible profiles with SupportedOperations enforcement (OperationSupportChecker). | Providers Overview · OpenAI-Compatible |
| Stage 1 Complexity Classifier | Heuristic prompt complexity scoring routes simple requests to fast/low-cost tiers and complex tasks to reasoning models automatically. | Routing & Load Balancing |
| Stage 2 Load Balancers | Six strategies across sibling deployments: weighted-shuffle, least-busy, latency-based (EWMA), usage-based-headroom, lowest-cost, and round-robin. | Routing & Load Balancing |
| 3-State Redis Circuit Breakers | Shared Closed → Open → Half-Open state in Redis across gateway replicas with atomic half-open probe leases. | Resiliency & Fallbacks |
| Bounded Fallback Chains | Configurable intra-group exponential retries and cross-model failover chains for 429/5xx/503 resilience. | Resiliency & Fallbacks |
| 2-Tier Exact-Match Cache | L1 in-process memory + L2 Redis SHA-256 exact-match response caching for sub-millisecond cache hits. | Exact-Match Caching |
Request Routing Pipeline
Explore Data Plane & Routing Guides
- Data Plane Endpoints: Chat Completions · Embeddings · Audio · Images · Rerank · Files · Batches
- Providers: Providers Overview · Custom Provider
- Routing & Resiliency: Routing & Load Balancing · Resiliency & Fallbacks · Exact-Match Caching