Skip to main content

System Architecture & Topology

GoRoute is engineered as an Enterprise AI Control Plane written in pure Go, organizing traffic routing, security governance, micro-dollar FinOps, observability, and agent management into five cohesive modules on top of a lock-free Go core and distributed persistence layer. It delivers sub-2ms hot-path routing overhead, multi-tenant isolation, and resilient upstream execution.

GoRoute Enterprise AI Control Plane (Pure Go) Go Core & Persistence Layer Clients, IDEs & Agents(OpenAI / Anthropic SDKs, Claude Code, Cursor, MCP, A2A) 1. AI Gateway(/v1/*, 73 Providers,EWMA, Circuit Breakers) 2. AI Governance(4-Layer RBAC, Vault,Guardrails, SSRF) 3. AI FinOps(Micro-Dollar Ledger,Outboxes, Lua Limits) 4. AI Observability(OTLP, Prometheus,Operator UI /ui/) 5. AI Agent Mgmt(MCP Hub, A2A gRPC,Tool Loops) L1 In-Memory Cache(Ring Buffers & Auth) Redis UniversalClient(Cluster/Sentinel, Lua, Pub/Sub) PostgreSQL 16+(Migrations 0001-0015, Outboxes)

GoRoute AI Gateway Architecture

[!NOTE] GoRoute's architecture eliminates synchronous PostgreSQL queries on the inference hot path using an L1 In-Memory + L2 Redis caching topology, single-roundtrip Redis Lua rate limiters, and asynchronous batched PostgreSQL spend pipelines.


Quick Installation & Operator Console​

You can install and spin up the complete GoRoute AI Gateway binary in seconds:

# 1. Download and install the precompiled binary
curl -fsSL https://raw.githubusercontent.com/goroute-ai/ai-gateway-go/main/install.sh | bash

# 2. Run the gateway with your configuration
ai-gateway --config config/gateway.yaml

Once running, access the embedded visual administration dashboard:

http://localhost:8080/ui/

The Operator UI lets you observe real-time inference latency, view active deployment health states, configure virtual keys, simulate guardrails, and run interactive streaming sessions.


The 10-Step Execution Lifecycle​

Every inference request traversing GoRoute undergoes a deterministic 10-step lifecycle:

Cache HIT Cache MISS Client Request(OpenAI SDK / Go SDK / cURL) 1. Auth and Vault(SHA-256 Virtual Key, 4-Layer RBAC) 2. Distributed Rate Limiter(Redis Lua Sliding Window RPM/TPM) 3. Exact Cache(Deterministic SHA-256)? 4. Return Cached Response(0.05ms Fast Path) 5. Stage 1 Heuristic Classifier(Sub-0.1ms Zero-LLM Fast vs Reasoning) 6. Stage 2 EWMA Load Balancer(6 Strategies) 7. 3-State Circuit Breakers(Healthy / Half-Open / Open) 8. Wire Protocol Normalization(Gemini, Claude, OpenAI, Mistral, Ollama) 9. Model Execution and SSE Stream 10. Asynchronous FinOps Ledger(Integer Micro-Dollars, pgx.Batch)

1. Dual-Boundary Authentication & Identity Resolution​

  • The incoming Bearer token (sk-gw-...) is hashed using SHA-256.
  • The token digest is verified against an in-memory L1 cache (15-second TTL) or Redis L2 cache.
  • The gateway resolves the caller's multi-tenant context: Organization ID, Team ID, User ID, Virtual Key scopes, and micro-dollar budget balances.
  • Outbound upstream provider credentials remain strictly isolated inside the AES-256-GCM encrypted credentials vault.

2. Distributed Rate Limiting & Quotas​

  • Executes an atomic Redis Lua script implementing a sliding-window counter.
  • Enforces both Requests Per Minute (RPM) and preflight estimated Tokens Per Minute (TPM).
  • If limits are reached, immediately returns HTTP 429 Too Many Requests with a precise Retry-After header.

3. Deterministic Response Caching​

  • Computes a deterministic SHA-256 cache key across {TenantID, Model, SystemPrompt, Messages, Tools}.
  • Deterministic queries (temperature: 0.0) resulting in a cache hit bypass the router and upstream providers entirely, returning in <0.05ms with zero token cost.

4. Stage 1 Heuristic Complexity Classification​

  • When the logical model alias model: "auto" is requested, GoRoute evaluates prompt complexity in under 0.1ms without invoking costly secondary LLMs:
    • Measures token volume and conversational turn depth.
    • Detects programming language keywords (func, class, import, SQL statements, backticks).
    • Scans for algorithmic, logical, and mathematical reasoning terms.
  • Simple tasks route to lightweight aliases (fast); complex tasks route to reasoning tiers (reasoning). Configure each alias with model IDs that are available to your provider account and exposed by the gateway catalog. The model behind an alias can change without changing the client request.

5. Stage 2 Load Balancing​

  • Selects the target provider deployment using one of six strategies:
    • weighted-shuffle: Proportional weighted random distribution.
    • least-busy: Routes to the deployment with the fewest active in-flight requests.
    • latency-based: Routes based on exponentially weighted moving average (EWMA) latency.
    • usage-based-headroom: Dynamically shifts traffic based on remaining RPM/TPM headroom.
    • lowest-cost: Prioritizes deployments with the lowest micro-dollar unit cost.
    • round-robin: Distributes requests evenly across deployments using a per-group atomic counter.

6. 3-State Circuit Breakers​

  • Tracks upstream health through a finite state machine: CLOSED (Healthy), OPEN (Tripped), and HALF_OPEN (Trial).
  • Three consecutive 5xx errors or network timeouts trip the circuit breaker, diverting subsequent requests immediately without waiting for timeouts.

7. Cross-Provider Fallback Cascade​

  • When an upstream fails, GoRoute automatically traverses configured fallback model groups (e.g., Primary OpenAI → Secondary Anthropic Claude → Tertiary Google Gemini).
  • Requests are retried with exponential backoff and decorrelated jitter.

8. Wire Protocol Normalization & Reasoning Adaptation​

  • Translates canonical gateway request schemas into vendor-specific payloads:
    • OpenAI: Maps developer/system messages and tool schemas.
    • Anthropic Claude: Consolidates parallel tool calls into strict alternating user/assistant turns and maps thinking token budgets.
    • Google Gemini: Converts message arrays into Gemini contents with thinkingConfig.
    • Ollama: Generates local JSON payloads.

9. Zero-Buffer SSE Streaming​

  • Server-Sent Events (SSE) emitted by upstream models are parsed on the fly.
  • Chunks are forwarded directly to the HTTP response flusher with zero memory buffering, maintaining real-time Time-To-First-Token (TTFT).

10. Asynchronous FinOps Micro-Dollar Ledger​

  • Token usage is priced using integer micro-dollars ($1.00 USD = 1,000,000 µ$) to eliminate floating-point rounding errors.
  • Spend events are placed onto a lock-free in-memory ring buffer.
  • A background worker flushes events to PostgreSQL in batched pgx.Batch transactions, keeping the data plane latency unaffected.

4-Layer Model Access Control Hierarchy​

GoRoute implements enterprise governance across 4 hierarchical evaluation layers:

[Layer 1: Global Model Catalogue]
│ (Discovered and registered provider models; availability varies by deployment)
▼
[Layer 2: Organization Subscriptions]
│ (Admin enables specific cloud providers: OpenAI, Anthropic, Gemini)
▼
[Layer 3: Organization Model Policies]
│ (Org-wide availability overrides and max token ceilings)
▼
[Layer 4: Virtual Key Allowlists]
│ (Scoped API key model permissions: e.g. ["fast", "gemini-2.5-flash"])
▼
[Execution Allowed]

[!IMPORTANT] If a model is not permitted at any of the 4 layers, GoRoute fails closed with an explicit HTTP 403 Forbidden or HTTP 404 Model Not Found.


Multi-Protocol Architecture​

GoRoute provides unified routing across three core protocols:

  1. OpenAI-Compatible Data Plane (/v1/*): Supports standard chat completions, embeddings, audio, images, files, batches, and realtime WebSockets.
  2. Model Context Protocol (/mcp/*, /mcp-rest/*): Serves as a central virtual hub aggregating distributed MCP tool servers with schema validation and progressive discovery.
  3. Agent-to-Agent Protocol (/a2a/v1/*): Provides RFC 8785 JCS canonicalized and RFC 7515 JWS signed Agent Cards, asynchronous task state machines, and native gRPC streaming.

Access all configured protocols and live health metrics via the Operator Web UI.