# GoRoute AI Control Plane: Complete Documentation Corpus
Generated from https://goroute-ai.github.io/
---
## GoRoute overview
URL: https://goroute-ai.github.io/docs/start/overview
Markdown: https://goroute-ai.github.io/docs/start/overview.md
Section: Control Plane Documentation > Start here
# GoRoute overview
GoRoute sits between an application and the AI services it uses. It gives an application a consistent way to send AI requests, while giving its team a place to choose providers, set access rules, and understand usage.
```mermaid
flowchart LR
app["Your application"] --> setup["Choose a GoRoute setup"] --> gateway["GoRoute gateway"] --> provider["AI provider"]
classDef appStyle fill:#153451,stroke:#60A5FA,color:#EFF6FF
classDef gatewayStyle fill:#0B5668,stroke:#22D3EE,color:#ECFEFF
classDef serviceStyle fill:#382F6B,stroke:#A78BFA,color:#F5F3FF
class app appStyle
class setup gatewayStyle
class gateway gatewayStyle
class provider serviceStyle
```
You can run GoRoute as a shared gateway or embed it directly in a Go application. The same product also includes tools for routing requests, protecting sensitive workflows, and connecting agents to approved tools.
## Choose a way to work
| What you want to do | Start here |
| :--- | :--- |
| Add the gateway inside a Go application | [Go SDK quickstart](https://goroute-ai.github.io/docs/start/quickstart-sdk) |
| Run a gateway that other applications can use | [Standalone gateway quickstart](https://goroute-ai.github.io/docs/start/quickstart-gateway) |
| Run the gateway with its database and cache | [Docker Compose quickstart](https://goroute-ai.github.io/docs/start/quickstart-docker) |
| Understand how requests move through the system | [Architecture and request flow](https://goroute-ai.github.io/docs/start/architecture) |
| Connect a coding tool or existing SDK | [Integrations overview](https://goroute-ai.github.io/docs/integrations/overview) |
## What happens to a request?
Your application sends a request to GoRoute. GoRoute checks the caller's access, applies the configured policies, chooses an available provider, and returns the provider's response. It can also record usage and trace information so your team can inspect what happened.
```mermaid
flowchart LR
App["Your application"] --> Access["Check access and policies"]
Access --> Route["Choose an available provider"]
Route --> Provider["AI provider"]
Provider --> Response["Return the response"]
Route --> Usage["Record usage and traces"]
```
## Choose a deployment style
| Deployment style | When it fits | Guide |
| :--- | :--- | :--- |
| **Standalone gateway** | Run GoRoute as a service that different applications can call. | [Gateway quickstart](https://goroute-ai.github.io/docs/start/quickstart-gateway) |
| **Docker Compose** | Try a local stack or run the gateway with PostgreSQL and Redis. | [Docker quickstart](https://goroute-ai.github.io/docs/start/quickstart-docker) |
| **Kubernetes** | Operate the gateway in a Kubernetes environment. | [Helm chart](https://goroute-ai.github.io/docs/deploy/helm-chart) · [Kustomize](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize) |
| **Embedded Go SDK** | Run gateway behavior in the same process as a Go application. | [Go SDK quickstart](https://goroute-ai.github.io/docs/start/quickstart-sdk) |
## Explore by goal
- [Choose and route model providers](https://goroute-ai.github.io/docs/gateway)
- [Protect requests and control access](https://goroute-ai.github.io/docs/governance)
- [Manage cost and request limits](https://goroute-ai.github.io/docs/finops)
- [Connect agents and tools](https://goroute-ai.github.io/docs/agents)
- [Deploy and operate GoRoute](https://goroute-ai.github.io/docs/deploy/configuration)
## Next steps
- [Run a gateway locally](https://goroute-ai.github.io/docs/start/quickstart-gateway)
- [Build with the Go SDK](https://goroute-ai.github.io/docs/start/quickstart-sdk)
- [Browse practical examples](https://goroute-ai.github.io/docs/examples/overview)
---
## Quickstart: Standalone Gateway CLI
URL: https://goroute-ai.github.io/docs/start/quickstart-gateway
Markdown: https://goroute-ai.github.io/docs/start/quickstart-gateway.md
Section: Control Plane Documentation > Start here
# Quickstart: Standalone Gateway CLI
This guide is for teams that want one shared gateway their applications can call. You will install GoRoute, configure a provider, start the service, and check that it responds. For an in-process setup inside a Go application, use the [Go SDK quickstart](https://goroute-ai.github.io/docs/start/quickstart-sdk).

> [!NOTE]
> GoRoute runs as a single static binary with no external interpreter dependencies. It includes an embedded Operator Web UI accessible directly at `http://localhost:8080/ui/`.
---
## 1. Quick Installation
### Option A: Universal Shell Installer (Recommended)
Install the compiled binary directly to `/usr/local/bin` (or `~/.local/bin`):
```bash
curl -fsSL https://raw.githubusercontent.com/goroute-ai/ai-gateway-go/main/install.sh | bash
```
To install a specific release version:
```bash
curl -fsSL https://raw.githubusercontent.com/goroute-ai/ai-gateway-go/main/install.sh | bash -s -- v1.8.0
```
Verify the installation:
```bash
ai-gateway -version
```
### Option B: Build from Source
If you have Go 1.26+ installed:
```bash
git clone https://github.com/goroute-ai/ai-gateway-go.git
cd ai-gateway-go
go build -o ai-gateway ./cmd/gateway
```
---
## 2. Configure the Gateway
Create a minimal configuration file `config.yaml` or use the provided template:
```yaml
server:
port: 8080
host: "0.0.0.0"
auth:
master_admin_key: "sk-gw-admin-secret-key-12345"
storage:
mode: "memory" # Use "postgres" for persistent multi-tenant production
providers:
openai:
api_key: "env.OPENAI_API_KEY"
anthropic:
api_key: "env.ANTHROPIC_API_KEY"
gemini:
api_key: "env.GEMINI_API_KEY"
ollama:
base_url: "http://localhost:11434"
model_groups:
- name: "fast"
routing_strategy: "weighted-shuffle"
deployments:
- provider: "gemini"
model: "gemini-2.5-flash"
weight: 60
- provider: "openai"
model: "gpt-4o-mini"
weight: 40
- name: "reasoning"
routing_strategy: "latency-based"
deployments:
- provider: "anthropic"
model: "claude-3-5-sonnet-20241022"
weight: 100
```
> [!TIP]
> Provider API keys can be passed directly as environment variables or referenced dynamically with `env.VARIABLE_NAME`.
---
## 3. Start the Gateway
Launch the service pointing to your configuration file:
```bash
./ai-gateway -config config.yaml
```
The gateway immediately initializes routes, background workers, and the embedded Operator Web UI on port `8080`.
---
## 4. Launch the Operator Web UI
Open your browser and navigate to:
```text
http://localhost:8080/ui/
```
When prompted, log in with your master admin key:
```text
sk-gw-admin-secret-key-12345
```
From the Operator UI, you can:
- Test real-time streaming completions in the **Chat Playground**.
- Monitor live token spend and request latencies in **Overview**.
- Inspect active upstream deployments and circuit breaker states in **Models & Routing**.
- Provision new Virtual API Keys for applications and developers in **Virtual Keys**.
---
## 5. Send Your First API Request
Send an inference request using standard OpenAI cURL format:
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-admin-secret-key-12345" \
-H "Content-Type: application/json" \
-d '{
"model": "fast",
"messages": [
{"role": "system", "content": "You are a concise Go systems engineer."},
{"role": "user", "content": "Explain how GoRoute manages sub-2ms routing overhead."}
],
"temperature": 0.7,
"max_tokens": 150
}'
```
### Inspect Gateway Response Headers
GoRoute returns rich telemetry and accounting headers on every request:
```http
HTTP/1.1 200 OK
Content-Type: application/json
X-Gateway-Request-Id: req_991efda4-7fab-4b03
X-Trace-Id: bad98e7deea34d7c810a963d9ed2c4ef
X-Gateway-Provider: gemini
X-Gateway-Model: gemini-2.5-flash
X-Gateway-Cost-Micros: 45
X-Gateway-Routing-Time: 12 µs
X-Gateway-Cache: MISS
```
---
## 6. Verify Service Health
Check process health probes:
```bash
# Liveness probe
curl http://localhost:8080/healthz
# Readiness probe (validates database and Redis connections)
curl http://localhost:8080/readyz
# Prometheus metrics
curl http://localhost:8080/metrics
```
---
## Next Steps
- Explore the complete [API Reference](https://goroute-ai.github.io/docs/api/gateway-api) for all 21 data plane endpoints.
- Configure [Virtual Keys & RBAC](https://goroute-ai.github.io/docs/governance/virtual-keys) for tenant management.
- Set up [Docker Compose](https://goroute-ai.github.io/docs/start/quickstart-docker) with PostgreSQL and Redis.
---
## Quickstart: Docker Compose
URL: https://goroute-ai.github.io/docs/start/quickstart-docker
Markdown: https://goroute-ai.github.io/docs/start/quickstart-docker.md
Section: Control Plane Documentation > Start here
# Quickstart: Docker Compose
Use this guide if you want to run GoRoute and its supporting services together in containers. The stack includes PostgreSQL for persistent data and Redis for shared rate limits and caching. If you only want to try the gateway locally, the [standalone quickstart](https://goroute-ai.github.io/docs/start/quickstart-gateway) is a lighter path.

> [!NOTE]
> If you prefer a lightweight single-binary host installation rather than containers, use the one-line shell installer:
> ```bash
> curl -fsSL https://raw.githubusercontent.com/goroute-ai/ai-gateway-go/main/install.sh | bash
> ```
---
## 1. Prerequisites
- Docker Engine 24.0+ and Docker Compose v2.20+.
- Valid API keys for upstream LLM providers (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`) or a local Ollama instance.
---
## 2. Clone and Inspect the Compose Stack
Clone the repository and inspect the deployment manifest:
```bash
git clone https://github.com/goroute-ai/ai-gateway-go.git
cd ai-gateway-go
```
The stack is defined in `deployments/docker-compose.yml`:
```yaml
version: "3.9"
services:
gateway:
build:
context: ..
dockerfile: deployments/Dockerfile
ports:
- "8080:8080"
environment:
- GATEWAY_MASTER_ADMIN_KEY=sk-gw-master-admin-default-key-change-me
- GATEWAY_STORAGE_MODE=postgres
- DATABASE_URL=postgres://goroute:goroute_secret@postgres:5432/goroute_db?sslmode=disable
- REDIS_URL=redis://redis:6379/0
- OPENAI_API_KEY=${OPENAI_API_KEY:-}
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-}
- GEMINI_API_KEY=${GEMINI_API_KEY:-}
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: goroute
POSTGRES_PASSWORD: goroute_secret
POSTGRES_DB: goroute_db
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U goroute -d goroute_db"]
interval: 5s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redisdata:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 5s
retries: 5
volumes:
pgdata:
redisdata:
```
---
## 3. Launch the Stack
Start the gateway, database, and cache containers in the background:
```bash
docker compose -f deployments/docker-compose.yml up -d --build
```
View startup logs to confirm automated database migrations and route registrations:
```bash
docker compose -f deployments/docker-compose.yml logs -f gateway
```
---
## 4. Launch the Operator Web UI
Once the containers are running and healthy, open the Operator Web UI in your browser:
```text
http://localhost:8080/ui/
```
Log in with the configured master admin key:
```text
sk-gw-master-admin-default-key-change-me
```
> [!IMPORTANT]
> The default development key `sk-gw-master-admin-default-key-change-me` must be replaced with a high-entropy secret before deploying to staging or production environments.
The Operator UI gives you complete control over:
- Live multi-provider chat playgrounds with streaming SSE.
- Real-time token consumption, request latencies, and micro-dollar FinOps balances.
- Automated health state machine inspections (`CLOSED`, `HALF_OPEN`, `OPEN`).
- Interactive guardrail policy simulation and PII redaction checks.
---
## 5. Send an Inference Request
Execute an OpenAI-compatible request against the running Docker stack:
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"model": "fast",
"messages": [
{"role": "user", "content": "Explain containerized microservices."}
],
"stream": false
}'
```
---
## 6. Verify Health and Distributed Readiness
Verify that both PostgreSQL and Redis are responsive through GoRoute's deep readiness check:
```bash
curl http://localhost:8080/readyz
```
Expected response (`HTTP 200 OK`):
```json
{
"status": "ready"
}
```
---
## 7. Teardown
To shut down the stack while preserving data volumes:
```bash
docker compose -f deployments/docker-compose.yml down
```
To purge persistent volumes as well:
```bash
docker compose -f deployments/docker-compose.yml down -v
```
---
## Next Steps
- Consult the [Operator UI Runbook](https://goroute-ai.github.io/docs/deploy/operator-ui) for administration workflows.
- Explore the [Admin Control Plane API](https://goroute-ai.github.io/docs/api/admin-api) for automated provisioning.
- Learn about [Distributed Rate Limiting](https://goroute-ai.github.io/docs/finops/rate-limiting) using Redis Lua scripts.
---
## Quickstart: Go SDK
URL: https://goroute-ai.github.io/docs/start/quickstart-sdk
Markdown: https://goroute-ai.github.io/docs/start/quickstart-sdk.md
Section: Control Plane Documentation > Start here
# Quickstart: Go SDK
Use this guide when you are building a Go application and want gateway behavior inside its process. It adds the SDK, creates a gateway, and sends a chat request. If your team already runs a shared gateway, the same guide also shows how to connect a Go client to it.
## Requirements
- Go 1.26 or newer.
- A provider API key for a live request. Without one, the example uses the SDK's mock provider option.
## 1. Add the SDK
From your Go module:
```bash
go get github.com/goroute-ai/ai-gateway-go/pkg/gateway@v1.8.0
```
Set a provider key if you want to send requests to a real provider:
```bash
export GEMINI_API_KEY="your-gemini-api-key"
```
## 2. Make a chat request
Create `main.go`:
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(gateway.ModelEntry{
ModelName: "fast",
Model: "gemini/gemini-2.5-flash",
APIKey: "os.environ/GEMINI_API_KEY",
}),
gateway.WithDemoMocksIfEnvMissing(),
)
if err != nil {
log.Fatal(err)
}
defer gw.Close()
result, err := gw.Chat(ctx, "fast", "Explain what an AI gateway does in one sentence.",
gateway.WithMaxTokens(128),
)
if err != nil {
log.Fatal(err)
}
fmt.Printf("[%s] %s\n", result.Provider, result.Content)
fmt.Printf("Cost: %d micro-dollars; routing: %d µs\n", result.CostMicros, result.RoutingMicros)
}
```
Run it:
```bash
go run .
```
With no `GEMINI_API_KEY`, `WithDemoMocksIfEnvMissing` supplies a local mock for this model. The mock is useful for trying the SDK flow; it does not verify provider credentials or live provider behavior.
## 3. Send a multi-turn request
The public SDK request type is `ChatRequest`; use `ChatWithMessages` for structured conversations:
```go
result, err := gw.ChatWithMessages(ctx, gateway.ChatRequest{
Model: "fast",
Messages: []gateway.Message{
{Role: "system", Content: "You are a concise Go tutor."},
{Role: "user", Content: "What does context cancellation do?"},
},
})
if err != nil {
log.Fatal(err)
}
fmt.Println(result.Content)
```
## 4. Stream a response
`StreamChat` invokes the callback as text deltas arrive and returns the aggregated result when the stream ends:
```go
_, err = gw.StreamChat(ctx, "fast", "Write a short haiku about Go.", func(delta string) error {
fmt.Print(delta)
return nil
})
if err != nil {
log.Fatal(err)
}
fmt.Println()
```
## Connect to a running gateway instead
Use `NewClient` when a gateway daemon is already running:
```go
client := gateway.NewClient("http://localhost:8080", "sk-gw-your-virtual-key")
result, err := client.Chat(ctx, "fast", "Hello from a remote Go client.")
```
## Next steps
- [SDK overview and execution modes](https://goroute-ai.github.io/docs/sdk/overview)
- [Client configuration](https://goroute-ai.github.io/docs/sdk/client-configuration)
- [CLI installation](https://goroute-ai.github.io/docs/start/quickstart-gateway)
- [Run the examples](https://goroute-ai.github.io/docs/examples/overview)
---
## System Architecture & Topology
URL: https://goroute-ai.github.io/docs/start/architecture
Markdown: https://goroute-ai.github.io/docs/start/architecture.md
Section: Control Plane Documentation > Start here
# System Architecture & Topology
GoRoute is engineered as an **Enterprise AI Control Plane** written in pure Go, organizing traffic routing, security governance, micro-dollar FinOps, observability, and agent management into five cohesive modules on top of a lock-free Go core and distributed persistence layer. It delivers sub-2ms hot-path routing overhead, multi-tenant isolation, and resilient upstream execution.
```mermaid
flowchart TB
Clients["Clients, IDEs & Agents
(OpenAI / Anthropic SDKs, Claude Code, Cursor, MCP, A2A)"] --> ControlPlane
subgraph ControlPlane["GoRoute Enterprise AI Control Plane (Pure Go)"]
direction LR
M1["1. AI Gateway
(/v1/*, 73 Providers,
EWMA, Circuit Breakers)"]
M2["2. AI Governance
(4-Layer RBAC, Vault,
Guardrails, SSRF)"]
M3["3. AI FinOps
(Micro-Dollar Ledger,
Outboxes, Lua Limits)"]
M4["4. AI Observability
(OTLP, Prometheus,
Operator UI /ui/)"]
M5["5. AI Agent Mgmt
(MCP Hub, A2A gRPC,
Tool Loops)"]
end
subgraph CorePersistence["Go Core & Persistence Layer"]
direction LR
L1["L1 In-Memory Cache
(Ring Buffers & Auth)"]
Redis["Redis UniversalClient
(Cluster/Sentinel, Lua, Pub/Sub)"]
PG["PostgreSQL 16+
(Migrations 0001-0015, Outboxes)"]
end
ControlPlane --> CorePersistence
```

> [!NOTE]
> GoRoute's architecture eliminates synchronous PostgreSQL queries on the inference hot path using an L1 In-Memory + L2 Redis caching topology, single-roundtrip Redis Lua rate limiters, and asynchronous batched PostgreSQL spend pipelines.
---
## Quick Installation & Operator Console
You can install and spin up the complete GoRoute AI Gateway binary in seconds:
```bash
# 1. Download and install the precompiled binary
curl -fsSL https://raw.githubusercontent.com/goroute-ai/ai-gateway-go/main/install.sh | bash
# 2. Run the gateway with your configuration
ai-gateway --config config/gateway.yaml
```
Once running, access the embedded visual administration dashboard:
```text
http://localhost:8080/ui/
```
The Operator UI lets you observe real-time inference latency, view active deployment health states, configure virtual keys, simulate guardrails, and run interactive streaming sessions.
---
## The 10-Step Execution Lifecycle
Every inference request traversing GoRoute undergoes a deterministic 10-step lifecycle:
```mermaid
graph TD
Client["Client Request
(OpenAI SDK / Go SDK / cURL)"] --> Auth["1. Auth and Vault
(SHA-256 Virtual Key, 4-Layer RBAC)"]
Auth --> RL["2. Distributed Rate Limiter
(Redis Lua Sliding Window RPM/TPM)"]
RL --> Cache{"3. Exact Cache
(Deterministic SHA-256)?"}
Cache -- "Cache HIT" --> Response["4. Return Cached Response
(0.05ms Fast Path)"]
Cache -- "Cache MISS" --> Router["5. Stage 1 Heuristic Classifier
(Sub-0.1ms Zero-LLM Fast vs Reasoning)"]
Router --> Balancer["6. Stage 2 EWMA Load Balancer
(6 Strategies)"]
Balancer --> CB["7. 3-State Circuit Breakers
(Healthy / Half-Open / Open)"]
CB --> Adapter["8. Wire Protocol Normalization
(Gemini, Claude, OpenAI, Mistral, Ollama)"]
Adapter --> Upstream["9. Model Execution and SSE Stream"]
Upstream --> FinOps["10. Asynchronous FinOps Ledger
(Integer Micro-Dollars, pgx.Batch)"]
FinOps --> Response
```
### 1. Dual-Boundary Authentication & Identity Resolution
- The incoming Bearer token (`sk-gw-...`) is hashed using SHA-256.
- The token digest is verified against an in-memory L1 cache (15-second TTL) or Redis L2 cache.
- The gateway resolves the caller's multi-tenant context: Organization ID, Team ID, User ID, Virtual Key scopes, and micro-dollar budget balances.
- Outbound upstream provider credentials remain strictly isolated inside the AES-256-GCM encrypted credentials vault.
### 2. Distributed Rate Limiting & Quotas
- Executes an atomic Redis Lua script implementing a sliding-window counter.
- Enforces both Requests Per Minute (RPM) and preflight estimated Tokens Per Minute (TPM).
- If limits are reached, immediately returns `HTTP 429 Too Many Requests` with a precise `Retry-After` header.
### 3. Deterministic Response Caching
- Computes a deterministic SHA-256 cache key across `{TenantID, Model, SystemPrompt, Messages, Tools}`.
- Deterministic queries (`temperature: 0.0`) resulting in a cache hit bypass the router and upstream providers entirely, returning in `<0.05ms` with zero token cost.
### 4. Stage 1 Heuristic Complexity Classification
- When the logical model alias `model: "auto"` is requested, GoRoute evaluates prompt complexity in under 0.1ms without invoking costly secondary LLMs:
- Measures token volume and conversational turn depth.
- Detects programming language keywords (`func`, `class`, `import`, SQL statements, backticks).
- Scans for algorithmic, logical, and mathematical reasoning terms.
- Simple tasks route to lightweight aliases (`fast`); complex tasks route to reasoning tiers (`reasoning`). Configure each alias with model IDs that are available to your provider account and exposed by the gateway catalog. The model behind an alias can change without changing the client request.
### 5. Stage 2 Load Balancing
- Selects the target provider deployment using one of six strategies:
- `weighted-shuffle`: Proportional weighted random distribution.
- `least-busy`: Routes to the deployment with the fewest active in-flight requests.
- `latency-based`: Routes based on exponentially weighted moving average (EWMA) latency.
- `usage-based-headroom`: Dynamically shifts traffic based on remaining RPM/TPM headroom.
- `lowest-cost`: Prioritizes deployments with the lowest micro-dollar unit cost.
- `round-robin`: Distributes requests evenly across deployments using a per-group atomic counter.
### 6. 3-State Circuit Breakers
- Tracks upstream health through a finite state machine: `CLOSED` (Healthy), `OPEN` (Tripped), and `HALF_OPEN` (Trial).
- Three consecutive 5xx errors or network timeouts trip the circuit breaker, diverting subsequent requests immediately without waiting for timeouts.
### 7. Cross-Provider Fallback Cascade
- When an upstream fails, GoRoute automatically traverses configured fallback model groups (e.g., Primary OpenAI → Secondary Anthropic Claude → Tertiary Google Gemini).
- Requests are retried with exponential backoff and decorrelated jitter.
### 8. Wire Protocol Normalization & Reasoning Adaptation
- Translates canonical gateway request schemas into vendor-specific payloads:
- **OpenAI**: Maps developer/system messages and tool schemas.
- **Anthropic Claude**: Consolidates parallel tool calls into strict alternating `user`/`assistant` turns and maps thinking token budgets.
- **Google Gemini**: Converts message arrays into Gemini `contents` with `thinkingConfig`.
- **Ollama**: Generates local JSON payloads.
### 9. Zero-Buffer SSE Streaming
- Server-Sent Events (SSE) emitted by upstream models are parsed on the fly.
- Chunks are forwarded directly to the HTTP response flusher with zero memory buffering, maintaining real-time Time-To-First-Token (TTFT).
### 10. Asynchronous FinOps Micro-Dollar Ledger
- Token usage is priced using integer micro-dollars ($1.00 USD = 1,000,000 µ$) to eliminate floating-point rounding errors.
- Spend events are placed onto a lock-free in-memory ring buffer.
- A background worker flushes events to PostgreSQL in batched `pgx.Batch` transactions, keeping the data plane latency unaffected.
---
## 4-Layer Model Access Control Hierarchy
GoRoute implements enterprise governance across 4 hierarchical evaluation layers:
```
[Layer 1: Global Model Catalogue]
│ (Discovered and registered provider models; availability varies by deployment)
▼
[Layer 2: Organization Subscriptions]
│ (Admin enables specific cloud providers: OpenAI, Anthropic, Gemini)
▼
[Layer 3: Organization Model Policies]
│ (Org-wide availability overrides and max token ceilings)
▼
[Layer 4: Virtual Key Allowlists]
│ (Scoped API key model permissions: e.g. ["fast", "gemini-2.5-flash"])
▼
[Execution Allowed]
```
> [!IMPORTANT]
> If a model is not permitted at any of the 4 layers, GoRoute fails closed with an explicit `HTTP 403 Forbidden` or `HTTP 404 Model Not Found`.
---
## Multi-Protocol Architecture
GoRoute provides unified routing across three core protocols:
1. **OpenAI-Compatible Data Plane (`/v1/*`)**: Supports standard chat completions, embeddings, audio, images, files, batches, and realtime WebSockets.
2. **Model Context Protocol (`/mcp/*`, `/mcp-rest/*`)**: Serves as a central virtual hub aggregating distributed MCP tool servers with schema validation and progressive discovery.
3. **Agent-to-Agent Protocol (`/a2a/v1/*`)**: Provides RFC 8785 JCS canonicalized and RFC 7515 JWS signed Agent Cards, asynchronous task state machines, and native gRPC streaming.
Access all configured protocols and live health metrics via the [Operator Web UI](http://localhost:8080/ui/).
---
## AI Gateway: Traffic Management & Model Routing
URL: https://goroute-ai.github.io/docs/gateway
Markdown: https://goroute-ai.github.io/docs/gateway.md
Section: Control Plane Documentation > Route AI requests
# AI Gateway: Traffic Management & Model Routing
The gateway is the traffic director between your application and its model providers. Your application makes requests through a consistent interface, and GoRoute handles provider selection, retries, and failover behind it. Experienced operators can also tune routing, caching, and operation-level compatibility.
```mermaid
flowchart LR
request["AI request"] --> router["GoRoute router"] --> provider["Selected provider"]
router --> fallback["Fallback provider"]
classDef appStyle fill:#153451,stroke:#60A5FA,color:#EFF6FF
classDef gatewayStyle fill:#0B5668,stroke:#22D3EE,color:#ECFEFF
classDef serviceStyle fill:#382F6B,stroke:#A78BFA,color:#F5F3FF
class request appStyle
class router gatewayStyle
class provider serviceStyle
class fallback serviceStyle
```
GoRoute currently provides **73 `ProviderType` profiles** (8 native adapters, 60+ built-in and extended OpenAI-compatible profiles, and `CustomOpenAICompatible`) through its unified API.
---
## Core Capabilities
| Capability | Implementation | Guide |
| :--- | :--- | :--- |
| **Unified Data Plane (24 `/v1` Paths)** | OpenAI-compatible `/v1/*` & `/openai/v1/*`, plus native Anthropic `/anthropic/v1/messages` & `count_tokens`, Responses API, Realtime WebSockets, `POST /v1/guardrails/apply`, Audio, Images, Files, Batches, Rerank, and Moderations. | [Chat Completions](https://goroute-ai.github.io/docs/gateway/endpoints/chat-completions) · [Messages](https://goroute-ai.github.io/docs/gateway/endpoints/messages) · [Responses](https://goroute-ai.github.io/docs/gateway/endpoints/responses) · [Realtime](https://goroute-ai.github.io/docs/gateway/endpoints/realtime) |
| **73+ Provider Profiles & `SupportedOperations`** | Native adapters for OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Gemini, Google Vertex, Cohere, Mistral, Ollama, plus 60+ OpenAI-compatible profiles with `SupportedOperations` enforcement (`OperationSupportChecker`). | [Providers Overview](https://goroute-ai.github.io/docs/providers/overview) · [OpenAI-Compatible](https://goroute-ai.github.io/docs/providers/openai-compatible) |
| **Stage 1 Complexity Classifier** | Heuristic prompt complexity scoring routes simple requests to fast/low-cost tiers and complex tasks to reasoning models automatically. | [Routing & Load Balancing](https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing) |
| **Stage 2 Load Balancers** | Six strategies across sibling deployments: `weighted-shuffle`, `least-busy`, `latency-based` (EWMA), `usage-based-headroom`, `lowest-cost`, and `round-robin`. | [Routing & Load Balancing](https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing) |
| **3-State Redis Circuit Breakers** | Shared `Closed → Open → Half-Open` state in Redis across gateway replicas with atomic half-open probe leases. | [Resiliency & Fallbacks](https://goroute-ai.github.io/docs/gateway/resiliency-and-fallbacks) |
| **Bounded Fallback Chains** | Configurable intra-group exponential retries and cross-model failover chains for 429/5xx/503 resilience. | [Resiliency & Fallbacks](https://goroute-ai.github.io/docs/gateway/resiliency-and-fallbacks) |
| **2-Tier Exact-Match Cache** | L1 in-process memory + L2 Redis SHA-256 exact-match response caching for sub-millisecond cache hits. | [Exact-Match Caching](https://goroute-ai.github.io/docs/gateway/caching) |
---
## Request Routing Pipeline
```mermaid
flowchart LR
Client["Client / SDK"] --> Cache["2-Tier Exact-Match Cache
(L1 Memory + L2 Redis)"]
Cache -->|Miss| Stage1["Stage 1: Complexity Classifier
(fast vs. reasoning alias)"]
Stage1 --> Stage2["Stage 2: Load Balancer
(weighted-shuffle / least-busy / latency-based /
headroom / lowest-cost / round-robin)"]
Stage2 --> CB["3-State Redis Circuit Breaker
& SupportedOperations Check"]
CB --> Provider["73+ Upstream Providers"]
Provider -->|429 / 5xx| Fallback["Bounded Retries &
Fallback Chain"]
Cache -->|Hit| Client
```
---
## Explore Data Plane & Routing Guides
- **Data Plane Endpoints**: [Chat Completions](https://goroute-ai.github.io/docs/gateway/endpoints/chat-completions) · [Embeddings](https://goroute-ai.github.io/docs/gateway/endpoints/embeddings) · [Audio](https://goroute-ai.github.io/docs/gateway/endpoints/audio) · [Images](https://goroute-ai.github.io/docs/gateway/endpoints/images) · [Rerank](https://goroute-ai.github.io/docs/gateway/endpoints/rerank) · [Files](https://goroute-ai.github.io/docs/gateway/endpoints/files) · [Batches](https://goroute-ai.github.io/docs/gateway/endpoints/batches)
- **Providers**: [Providers Overview](https://goroute-ai.github.io/docs/providers/overview) · [Custom Provider](https://goroute-ai.github.io/docs/providers/custom-provider)
- **Routing & Resiliency**: [Routing & Load Balancing](https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing) · [Resiliency & Fallbacks](https://goroute-ai.github.io/docs/gateway/resiliency-and-fallbacks) · [Exact-Match Caching](https://goroute-ai.github.io/docs/gateway/caching)
---
## Routing & Load Balancing
URL: https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing
Markdown: https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing.md
Section: Control Plane Documentation > Route AI requests
# Routing & Load Balancing
GoRoute combines a **Stage 1 zero-LLM heuristic complexity classifier** (`model: "auto"`) with **Stage 2 lock-free load balancing** across model deployments so every request lands on the optimal upstream tier with sub-millisecond routing overhead.
---
## Stage 1: Zero-LLM Heuristic Complexity Classifier
When clients request `model: "auto"`, GoRoute AI Gateway evaluates the prompt using a zero-LLM heuristic complexity classifier that executes in **under 0.1 milliseconds (< 100 µs)** without invoking any auxiliary LLMs or consuming external API tokens.
```mermaid
flowchart LR
Prompt["Incoming Prompt
(model: 'auto')"] --> Stage1["Stage 1 Heuristic Classifier
(Sub-0.1ms Execution)"]
Stage1 --> Signals{"Complexity Signals:
Context Length, Code Markers,
Math & Multi-Step Reasoning"}
Signals -->|"Score < 0.40"| Fast["Fast Tier Alias ('fast')
(Gemini 2.5 Flash, GPT-4o-mini)
Low Cost, 50-100 tok/s"]
Signals -->|"Score >= 0.40"| Reasoning["Reasoning Tier Alias ('reasoning')
(Claude Sonnet 4, o1, o3-mini)
Deep Analysis, Advanced Math/Code"]
```
### Classification Scoring Model
The classifier computes a normalized complexity score between `0.0` and `1.0` based on 4 primary signal vectors:
```text
Complexity Score = (w_tokens * S_tokens) + (w_code * S_code) + (w_math * S_math) + (w_reasoning * S_reasoning)
```
1. **Context & Token Volume (`S_tokens`)**: Evaluates conversation history length, message turn count, and token volume against configurable thresholds (default: `long_context_tokens: 32000`).
2. **Code & Syntax Indicators (`S_code`)**: Detects triple-backtick markdown blocks, programming keywords (`func`, `class`, `import`, `def`, `interface`), stack traces, and SQL schemas.
3. **Mathematical & Algorithmic Markers (`S_math`)**: Identifies equations, statistical formulas, and computational notation.
4. **Multi-Step Reasoning Keywords (`S_reasoning`)**: Scans for semantic reasoning triggers such as *"prove step by step"*, *"distributed consensus"*, *"edge cases"*, and *"formal verification"*.
### Classifier Configuration (`configs/config.yaml`)
```yaml
router:
default_fast_alias: "fast"
default_reasoning_alias: "reasoning"
default_coding_alias: "coding"
complexity_threshold: 0.40 # Prompts >= 0.40 route to reasoning tier
long_context_tokens: 32000 # Prompts exceeding 32k tokens route to reasoning
```
> [!NOTE]
> `model: "auto"` runs the two-tier `classifyPrompt` cascade (Tier 1 deterministic `long_context_tokens` / `reasoning_effort` check, followed by the Tier 2 multi-feature scorer) and routes between `default_fast_alias` (`"fast"`) and `default_reasoning_alias` (`"reasoning"`). Requesting `"coding"` directly resolves the configured `coding` model group by alias.
### Benchmark Overhead
| Evaluation Method | Added Latency | Token Cost | Reliability |
| :--- | :--- | :--- | :--- |
| **GoRoute Stage 1 Heuristic** | **< 0.012 ms (12 µs)** | **$0.00 (Zero tokens)** | **100% Deterministic** |
| LLM-as-a-Router (e.g. GPT-4o-mini classifier) | 350 ms – 700 ms | $0.0002 per prompt | Subject to hallucination / timeouts |
| Embedding Distance Classifier | 45 ms – 90 ms | Vector search lookup overhead | Requires embeddings vector database |
---
## Stage 2: Load Balancing Strategies
Once a model group or alias is resolved, the load balancer selects an active upstream deployment using one of **six** `domain.RoutingStrategy` values (`weighted-shuffle`, `least-busy`, `latency-based`, `usage-based-headroom`, `lowest-cost`, `round-robin`). Note that the latency strategy constant is **`latency-based`** (not `lowest-latency`). When Redis is configured, adaptive strategies (`least-busy`, `latency-based`, `usage-based-headroom`) read shared telemetry across pods.
```mermaid
graph TD
Alias["Logical Alias ('fast')"] --> LB["Stage 2 Load Balancer"]
LB --> Strat1["1. weighted-shuffle"]
LB --> Strat2["2. least-busy"]
LB --> Strat3["3. latency-based (EWMA)"]
LB --> Strat4["4. usage-based-headroom"]
LB --> Strat5["5. lowest-cost"]
LB --> Strat6["6. round-robin"]
Strat1 --> UpstreamPool["Deployment Pool (Gemini Flash, GPT-4o-mini, Mistral)"]
```
### Selection Algorithms
#### 1. `weighted-shuffle`
Selects deployments randomly in proportion to their configured integer weights. A weight of zero or less is treated as the default weight of 100.
#### 2. `least-busy`
Tracks active in-flight requests (including Redis-shared reservation signals in multi-replica deployments) and selects a deployment with the smallest observed count.
#### 3. `latency-based` (EWMA)
Computes an exponentially weighted moving average of observed response latency:
```text
EWMA_t = alpha * Latency_t + (1 - alpha) * EWMA_(t-1)
```
The deployment with the lowest observed moving average is preferred. New deployments use a default baseline until latency has been observed.
#### 4. `usage-based-headroom`
Estimates remaining token capacity from request and token usage tracked by the gateway. It does not use this strategy as a guarantee against provider rate limits.
#### 5. `lowest-cost`
Selects the candidate with the lowest sum of configured input and output pricing values in the model catalogue. This does not evaluate every request parameter or provider's live price; keep catalogue data current.
#### 6. `round-robin`
Cycles deterministically across healthy candidate deployments using a lock-free per-`ModelGroupID` atomic counter (`selectRoundRobin`). Useful for uniform distribution across homogeneous endpoints when weights or adaptive telemetry are not needed.
### YAML Configuration
```yaml
model_list:
- model_name: fast
routing_strategy: weighted-shuffle
model_params:
model: gemini/gemini-2.5-flash
weight: 80
- model_name: fast
routing_strategy: weighted-shuffle
model_params:
model: openai/gpt-4o-mini
weight: 20
- model_name: compute-heavy
routing_strategy: least-busy
model_params:
model: anthropic/claude-sonnet-4-20250514
```
---
## Resiliency, Circuit Breakers & Fallbacks
URL: https://goroute-ai.github.io/docs/gateway/resiliency-and-fallbacks
Markdown: https://goroute-ai.github.io/docs/gateway/resiliency-and-fallbacks.md
Section: Control Plane Documentation > Route AI requests
# Resiliency, Circuit Breakers & Fallbacks
GoRoute AI Gateway protects inference availability during regional outages, HTTP 429 rate limits, and upstream 5xx failures by pairing **3-state circuit breakers** with **jittered intra-group retries** and **cross-provider fallback cascades**.
---
## 1. 3-State Circuit Breakers & Outage Isolation
GoRoute embeds a state machine based on the classic **3-State Circuit Breaker Pattern** (`healthy`, `open`, `half-open`) to automatically isolate unhealthy upstream deployments and prevent cascading failures.
```mermaid
stateDiagram-v2
[*] --> Healthy
Healthy --> Open: Consecutive Failures >= failure_threshold (3)
Open --> HalfOpen: Cooldown Timer Expires (60s)
HalfOpen --> Healthy: Probe Request Succeeds (200 OK)
HalfOpen --> Open: Probe Request Fails (5xx / Timeout)
```
### State Transitions
1. **`Healthy` (Closed)**: Normal operational state. All traffic passes through to the selected deployment. Successes and latencies are tracked for EWMA calculations.
2. **`Open` (Tripped)**: When a deployment accumulates `failure_threshold` (default: 3) consecutive 5xx errors or timeouts within `failure_window` (default: 30s), the breaker trips to **OPEN**.
- All subsequent requests targeting this deployment are **immediately diverted** to the next deployment in the fallback chain without attempting network I/O.
- The deployment enters a quarantine cooldown (default: `default_cooldown: 60s`).
3. **`Half-Open` (Probing)**: When the cooldown timer expires, the breaker transitions to **HALF-OPEN**.
- A single canary request is allowed through to test upstream health.
- If the probe succeeds with HTTP 200, the breaker resets to **HEALTHY** and resumes full traffic.
- If the probe fails, the breaker trips back to **OPEN** and resets the quarantine cooldown.
### Cluster-Wide Redis Coordination
When running multiple gateway replicas behind a load balancer with Redis configured:
- **Shared Failure State**: Deployment failure counts and open states are synchronized via Redis so all replicas isolate an unhealthy upstream together.
- **Half-Open Probe Leases**: Replicas coordinate half-open canary probes through a Redis lease so only a single pod sends a probe request when cooldown expires, preventing probe storms.
### Circuit Breaker Configuration (`configs/config.yaml`)
```yaml
router:
failure_threshold: 3 # Consecutive errors required to trip breaker
failure_window: 30s # Sliding time window for failure counts
default_cooldown: 60s # Quarantine duration before half-open probe
```
---
## 2. Retries & Cross-Provider Fallback Cascades
When cloud AI providers experience regional outages, rate limits (HTTP 429), or internal server errors (HTTP 500, 502, 503), GoRoute AI Gateway automatically executes intra-group retries and cross-provider fallback cascades.
```mermaid
sequenceDiagram
participant Client
participant GW as GoRoute AI Gateway
participant OpenAI as Primary (GPT-4o)
participant Claude as Backup (Claude 3.5)
Client->>GW: 1. POST /v1/chat/completions (model: "prod-chat")
Note over GW: Attempt 1: Route to Primary
GW->>OpenAI: 2. Forward Request
OpenAI-->>GW: 3. HTTP 503 Service Unavailable
Note over GW: Jittered Backoff (50ms)
GW->>OpenAI: 4. Intra-Group Retry #2
OpenAI-->>GW: 5. HTTP 503 Service Unavailable
Note over GW: Trip Circuit Breaker & Failover
GW->>Claude: 6. Fallback Request
Claude-->>GW: 7. HTTP 200 OK (Stream)
GW-->>Client: 8. HTTP 200 OK (X-Gateway-Fallback: true)
```
### Intra-Group Retries with Full Jitter
Before abandoning a model deployment, GoRoute executes automated retries with exponential backoff and randomized decorrelated jitter to prevent thundering herd spikes on recovered upstreams:
```text
Delay = min(max_retry_delay, base_retry_delay * 2^attempt) * Uniform(0.5, 1.5)
```
```yaml
router:
default_max_retries: 2
base_retry_delay: 50ms
max_retry_delay: 2s
```
- **Streaming Pre-First-Token Retries**: Streaming endpoints (`/v1/chat/completions`, `/v1/messages`, `/v1/responses`) apply exponential backoff retries if an upstream fails *before* emitting the first output token, then stream directly once bytes begin flowing.
- **Multi-Modal Reliability**: `ExecuteEmbeddings` (`/v1/embeddings`) and audio/image endpoints share the same retry, circuit-breaker, and fallback pipeline as chat completions.
- **Bounded Fallback Graph Traversal**: Fallback group chains enforce explicit cycle detection and maximum depth limits to prevent infinite failover loops.
### Declarative Cross-Provider Fallback Chains
Configure fallback cascades in `configs/config.yaml`:
```yaml
model_list:
# Primary Group: OpenAI GPT-4o
- model_name: prod-chat
routing_strategy: weighted-shuffle
model_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
fallbacks: ["backup-claude", "backup-gemini"]
# Secondary Fallback: Anthropic Claude Sonnet
- model_name: backup-claude
model_params:
model: anthropic/claude-sonnet-4-20250514
api_key: os.environ/ANTHROPIC_API_KEY
# Tertiary Emergency Fallback: Google Gemini Flash
- model_name: backup-gemini
model_params:
model: gemini/gemini-2.5-flash
api_key: os.environ/GEMINI_API_KEY
```
If OpenAI experiences an outage, GoRoute seamlessly falls back to Anthropic Claude. If Anthropic also fails, traffic transparently cascades to Google Gemini Flash.
---
## Deterministic Exact-Match Cache
URL: https://goroute-ai.github.io/docs/gateway/caching
Markdown: https://goroute-ai.github.io/docs/gateway/caching.md
Section: Control Plane Documentation > Route AI requests
# Deterministic Exact-Match Cache
GoRoute AI Gateway embeds a tenant-isolated, deterministic exact-match response cache that serves identical queries in **0.05 milliseconds** with **zero upstream token spend**.
```mermaid
graph TD
Request["Incoming Chat Request"] --> CheckTemp{"temperature == 0.0?"}
CheckTemp -- "No (stochastic)" --> Upstream["Direct Upstream Inference"]
CheckTemp -- "Yes (deterministic)" --> KeyGen["Compute SHA-256 Digest:
SHA256(OrgID + Model + System + Messages + Tools)"]
KeyGen --> Redis{"Cache HIT in Redis / Memory?"}
Redis -- "HIT" --> FastResponse["Return Cached Response
(0.05ms, $0.00 Cost, X-Gateway-Cache: HIT)"]
Redis -- "MISS" --> Upstream
Upstream --> Store["Store Response in Cache (TTL: 1h)"]
```
---
## 1. Deterministic Hashing Key
To guarantee tenant isolation and output correctness, the cache key is constructed using a cryptographic SHA-256 digest over all canonical output-affecting request fields (organization, model, messages, system prompt, tools, sampling/output parameters, and `extra_body`):
```text
CacheKey = SHA256(OrgID || Epoch || CanonicalRequestFields)
```
- **Tenant Isolation**: An identical prompt submitted by Org A will never be returned to Org B.
- **Cross-Replica Invalidation Epochs**: Calling `POST /admin/v1/cache/invalidate` increments a Redis-backed organization invalidation epoch (`{org:}` hash-tagged) so cached entries are invalidated immediately across all gateway replicas.
- **Deterministic Only**: Requests with `temperature > 0.0` or non-zero randomness bypass the cache to preserve LLM creativity and variance.
---
## 2. Benchmark Comparison
| Metric | Upstream Model Call | GoRoute Cached Hit |
| :--- | :--- | :--- |
| **Response Latency** | 450 ms – 1,800 ms | **0.05 ms (50 µs)** |
| **Cost Incurred** | Full token billing ($0.001 – $0.05) | **$0.00 (Zero tokens)** |
| **Upstream Network I/O** | 100% | **0% (Bypassed)** |
| **Telemetry Header** | `X-Gateway-Cache: MISS` | `X-Gateway-Cache: HIT` |
---
## Chat Completions API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/chat-completions
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/chat-completions.md
Section: Control Plane Documentation > Use the API
# Chat Completions API
The GoRoute Data Plane provides a 100% OpenAI-compatible Chat Completions endpoint standardizing request parameters, tool calls, and real-time SSE streaming across all upstream providers (OpenAI, Anthropic Claude, Google Gemini, Mistral, Cerebras, Groq, DeepSeek, and local Ollama models).
---
## Endpoint Specification
- **Method**: `POST`
- **Path**: `/v1/chat/completions` (also supports unversioned `/chat/completions`)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
---
## Key Capabilities & Enterprise Extensions
### 1. Dynamic Provider & Model Routing
Beyond predefined logical aliases (`fast`, `reasoning`, `coding`), GoRoute supports direct dynamic wildcard dispatch via `provider/model` syntax:
```json
{
"model": "cerebras/llama-3.3-70b",
"messages": [{"role": "user", "content": "Explain kernel bypass networking."}]
}
```
Other direct provider routing examples:
- `anthropic/claude-3-5-sonnet`
- `gemini/gemini-2.5-flash`
- `groq/llama-3.3-70b-versatile`
- `deepseek/deepseek-chat`
> [!NOTE]
> Direct provider routing evaluates your organization's Layer 2 Provider Subscriptions, Layer 3 Org Model Policies, and Virtual Key allowlists before dynamically provisioning or dispatching to the target provider.
### 2. Reasoning Effort & Thinking Budgets
For reasoning models (OpenAI o1/o3/o4, Anthropic Claude 3.7 Sonnet, Google Gemini 2.5), GoRoute accepts standard `reasoning_effort`:
```json
{
"model": "reasoning",
"messages": [{"role": "user", "content": "Solve this distributed systems puzzle."}],
"reasoning_effort": "high"
}
```
The gateway standardizes reasoning across providers:
- **Anthropic Claude**: Maps `"low"`, `"medium"`, and `"high"` to explicit `thinking: {"type": "enabled", "budget_tokens": N}` (`1024`, `4096`, or `16384` tokens).
- **Google Gemini 2.5**: Injects `thinkingConfig.thinkingBudget` accordingly.
- **OpenAI o1/o3**: Passes native `reasoning_effort`.
- **Pre-Flight Estimation**: Reserves an additional 1,024 tokens during pre-flight rate limiting when `reasoning_effort` is detected.
### 3. Strict Role Alternation & Tool Consolidation
Anthropic and Google Gemini mandate strict turn alternation between `user` and `assistant`. In multi-tool calling flows, OpenAI returns multiple consecutive `role: "tool"` messages for parallel calls.
When routing to Anthropic or Gemini, GoRoute automatically **consolidates consecutive tool results** into a single `user` turn with multiple `tool_result` content blocks. You never need to hand-craft vendor-specific turn structures.
---
## Code Examples
#### cURL
```bash
# 1. Synchronous Chat Completion with direct provider routing
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "cerebras/llama-3.3-70b",
"messages": [
{"role": "system", "content": "You are a concise systems architect."},
{"role": "user", "content": "Why is DPDK faster than traditional kernel socket I/O?"}
],
"temperature": 0.2,
"max_tokens": 256
}'
# 2. Hybrid Reasoning with thinking effort on Claude 3.7
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-3-5-sonnet",
"messages": [
{"role": "user", "content": "Prove that Raft election safety holds in asynchronous networks."}
],
"reasoning_effort": "high"
}'
# 3. Server-Sent Events (SSE) Streaming
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "fast",
"messages": [{"role": "user", "content": "Stream a haiku about distributed consensus."}],
"stream": true
}'
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
# Synchronous with reasoning effort
response = client.chat.completions.create(
model="anthropic/claude-3-5-sonnet",
messages=[{"role": "user", "content": "Explain zero-copy buffer pooling in Go."}],
extra_body={"reasoning_effort": "medium"}
)
print(response.choices[0].message.content)
# Zero-buffer streaming
stream = client.chat.completions.create(
model="cerebras/llama-3.3-70b",
messages=[{"role": "user", "content": "Stream me the steps of TLS 1.3 handshake."}],
stream=True
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")
```
#### Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{ModelName: "fast", Model: "gemini/gemini-2.5-flash"},
gateway.ModelEntry{ModelName: "reasoning", Model: "anthropic/claude-3-5-sonnet"},
),
gateway.WithDemoMocksIfEnvMissing(),
)
if err != nil {
log.Fatal(err)
}
defer gw.Close()
// Synchronous completion
res, err := gw.Chat(ctx, "fast", "Explain zero-allocation design in Go.")
if err != nil {
log.Fatal(err)
}
fmt.Printf("[%s] %s (cost: %d µ$)\n", res.Model, res.Content, res.CostMicros)
// Zero-buffer SSE streaming
_, err = gw.StreamChat(ctx, "fast", "Stream a 1-sentence haiku.", func(delta string) error {
fmt.Print(delta)
return nil
})
}
```
#### TypeScript / Node
```typescript
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "sk-gw-prod-7f9a2b8c4d1e",
baseURL: "http://localhost:8080/v1",
});
async function run() {
const stream = await client.chat.completions.create({
model: "cerebras/llama-3.3-70b",
messages: [{ role: "user", content: "Stream me a summary of Raft consensus." }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content || "");
}
}
run();
```
---
## Gateway Telemetry Response Headers
Every chat completion response includes detailed telemetry headers for end-to-end tracing and auditing:
| Header | Description | Example |
| :--- | :--- | :--- |
| `X-Gateway-Request-Id` | Unique UUID assigned to the request lifecycle | `req_fb502e5e-b39a-4d9a` |
| `X-Trace-Id` | W3C / OpenTelemetry distributed trace ID | `8d380107e03c29fd8d979bbc660a10e9` |
| `X-Gateway-Provider` | The winning upstream provider selected | `cerebras`, `anthropic`, `gemini` |
| `X-Gateway-Model` | The physical upstream model executed | `llama-3.3-70b`, `claude-3-5-sonnet` |
| `X-Gateway-Cost-Micros` | Micro-dollar spend incurred ($0.000001 USD) | `13` ($0.000013 USD) |
| `X-Gateway-Routing-Time` | Gateway Stage 1 + Stage 2 routing overhead | `12 µs` |
| `X-Gateway-Cache` | Cache evaluation result | `HIT` or `MISS` |
| `Retry-After` | Cooldown period sent on HTTP 429 rate limit | `45` (seconds) |
> [!TIP]
> When `X-Gateway-Cache: HIT` is returned, `X-Gateway-Cost-Micros` is `0`, confirming that the response was served from cache without upstream provider token charges.
---
## Anthropic Messages Ingress API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/messages
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/messages.md
Section: Control Plane Documentation > Use the API
# Anthropic Messages Ingress API
GoRoute AI Gateway provides native wire compatibility with the **Anthropic Messages API**. Clients and agent frameworks built against the official Anthropic SDK (`@anthropic-ai/sdk` in TypeScript, `anthropic` in Python) can point directly to GoRoute without code changes.
The gateway transparently translates between Anthropic's message format and canonical representation, handles thinking blocks, normalizes tool calling schemas, and provides a zero-latency token counter at `/v1/messages/count_tokens`.
---
## Endpoint Specification
- **HTTP Method**: `POST`
- **Supported Paths**:
- `/v1/messages` (Standard gateway ingress)
- `/messages` (Unversioned compatibility alias)
- `/anthropic/v1/messages` (Direct Anthropic wire spec alias)
- `/v1/messages/count_tokens` (Pre-flight token counting)
- `/anthropic/v1/messages/count_tokens` (Anthropic count_tokens wire spec)
- **Authentication**: `Bearer sk-gw-...` or `x-api-key: sk-gw-...`
- **Content-Type**: `application/json`
- **Error Format**: Errors on Anthropic routes return native Anthropic error envelopes (`{"type":"error","error":{"type":"...","message":"..."}}`).
---
## Architecture & Turn Translation
```mermaid
sequenceDiagram
participant Client
participant GW as GoRoute AI Gateway
participant LLM as Upstream LLM (Claude)
Client->>GW: 1. POST /v1/messages (Anthropic Spec)
Note over GW: Extract System Blocks & Thinking Budget
Note over GW: Consolidate Alternating Turns & Schema
GW->>LLM: 2. Canonical Inference Dispatch
LLM-->>GW: 3. Canonical Response / Stream
Note over GW: Translate to Anthropic Event Model
GW-->>Client: 4. Anthropic Format Response
```
### Key Translation Mechanics
1. **System Prompt Normalization**:
Anthropic requires system instructions to be passed via a top-level `system` field (either a string or an array of text blocks). In the OpenAI / Canonical format, system instructions are injected as messages with `role: "system"`. GoRoute automatically extracts and reconciles these representations.
2. **Strict Alternating Turn Consolidation**:
Anthropic mandates that messages in the `messages` array strictly alternate between `user` and `assistant`. In multi-tool calling flows:
- When tools execute in parallel, OpenAI returns multiple consecutive `role: "tool"` messages.
- GoRoute automatically consolidates consecutive tool results into a single `user` turn containing multiple `tool_result` content blocks, preserving Anthropic contract compliance.
3. **Thinking Blocks (`extended_thinking`)**:
Anthropic Claude 3.7 Sonnet supports hybrid reasoning via `thinking: {"type": "enabled", "budget_tokens": 4096}`. GoRoute preserves thinking blocks, calculates reasoning token pricing, and streams thought traces seamlessly.
4. **Tool Use & Schema Translation**:
Anthropic uses `input_schema` inside tool definitions, whereas OpenAI uses `parameters`. GoRoute dynamically bi-directionally maps between `input_schema` and `parameters`, as well as between `tool_use` / `tool_result` blocks and canonical tool calls.
---
## Request & Response Schemas
### Request Schema (`anthropicMessagesPayload`)
```json
{
"model": "claude-3-7-sonnet",
"max_tokens": 1024,
"system": "You are an expert distributed systems engineer.",
"messages": [
{
"role": "user",
"content": "Analyze Paxos vs Raft consensus."
}
],
"thinking": {
"type": "enabled",
"budget_tokens": 2048
},
"tools": [
{
"name": "lookup_cluster_state",
"description": "Returns current Raft leader and term",
"input_schema": {
"type": "object",
"properties": {
"cluster_id": {"type": "string"}
},
"required": ["cluster_id"]
}
}
]
}
```
### Response Schema
```json
{
"id": "msg_c9a81230-1049-411a-829d-47291a90f12a",
"type": "message",
"role": "assistant",
"model": "claude-3-7-sonnet",
"content": [
{
"type": "text",
"text": "Both Paxos and Raft provide equivalent state machine replication safety..."
}
],
"stop_reason": "end_turn",
"stop_sequence": null,
"usage": {
"input_tokens": 34,
"output_tokens": 112
}
}
```
---
## Server-Sent Events (SSE) Streaming Format
When `"stream": true` is passed, GoRoute emits Anthropic-compliant event streams:
```http
event: message_start
data: {"type":"message_start","message":{"id":"msg_...","type":"message","role":"assistant","model":"claude-3-7-sonnet","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":34,"output_tokens":0}}}
event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Paxos"}}
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" and Raft"}}
event: content_block_stop
data: {"type":"content_block_stop","index":0}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":112}}
event: message_stop
data: {"type":"message_stop"}
```
---
## Token Counting Endpoint (`/v1/messages/count_tokens`)
GoRoute provides a zero-latency, in-process token counter that calculates the exact input token requirements for an Anthropic message payload prior to sending it to the model.
### Request
```bash
curl -X POST http://localhost:8080/v1/messages/count_tokens \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-3-7-sonnet",
"system": "You are a concise code assistant.",
"messages": [
{"role": "user", "content": "How do goroutines manage stack growth?"}
]
}'
```
### Response
```json
{
"input_tokens": 28
}
```
> [!TIP]
> The count_tokens endpoint calculates subword tokens using GoRoute's internal BPE tokenizer profile (`claude_bpe`, 3.6 chars/token) plus per-message framing overhead, executing in under 50 microseconds without network egress.
---
## Code Examples
#### Python (Anthropic SDK)
```python
import anthropic
client = anthropic.Anthropic(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080"
)
# Standard Messages call
message = client.messages.create(
model="claude-3-7-sonnet",
max_tokens=256,
system="You are a helpful software architect.",
messages=[
{"role": "user", "content": "Explain memory pooling in Go."}
]
)
print(message.content[0].text)
# Token counting
count = client.messages.count_tokens(
model="claude-3-7-sonnet",
messages=[
{"role": "user", "content": "Pre-flight token estimation check."}
]
)
print(f"Estimated input tokens: {count.input_tokens}")
```
#### cURL
```bash
# Messages request via /anthropic/v1/messages
curl -X POST http://localhost:8080/anthropic/v1/messages \
-H "x-api-key: sk-gw-prod-7f9a2b8c4d1e" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-3-7-sonnet",
"max_tokens": 512,
"messages": [
{"role": "user", "content": "What is the function of sync.Pool?"}
]
}'
```
#### TypeScript (Anthropic SDK)
```typescript
import Anthropic from "@anthropic-ai/sdk";
const anthropic = new Anthropic({
apiKey: "sk-gw-prod-7f9a2b8c4d1e",
baseURL: "http://localhost:8080",
});
async function main() {
const stream = await anthropic.messages.stream({
model: "claude-3-7-sonnet",
max_tokens: 256,
messages: [{ role: "user", content: "Tell me about Go channels." }],
});
for await (const event of stream) {
if (
event.type === "content_block_delta" &&
event.delta.type === "text_delta"
) {
process.stdout.write(event.delta.text);
}
}
}
main();
```
---
## Legacy Text Completions API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/completions
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/completions.md
Section: Control Plane Documentation > Use the API
# Legacy Text Completions API
While modern generative LLM architectures predominantly utilize the conversational `/v1/chat/completions` schema, GoRoute AI Gateway maintains first-class support for the classic **Text Completions API** (`POST /v1/completions`). This endpoint is essential for base foundation models, instruction-following base models (such as `gpt-3.5-turbo-instruct`), code-infilling engines, and legacy pipelines.
---
## Endpoint Specification
- **HTTP Method**: `POST`
- **Supported Paths**:
- `/v1/completions`
- `/completions` (Unversioned alias)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
---
## Request & Response Schemas
### Request Schema (`CanonicalCompletionRequest`)
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `model` | string | **Yes** | Target model (e.g. `gpt-3.5-turbo-instruct`, `mistral-tiny`). |
| `prompt` | string \| string[] | **Yes** | The prompt(s) to generate completions for. Can be a single string or an array of strings. |
| `max_tokens` | integer | No | Maximum number of tokens to generate. |
| `temperature` | float | No | Sampling temperature between `0.0` and `2.0`. |
| `top_p` | float | No | Nucleus sampling probability cutoff (alternative to temperature). |
| `stream` | boolean | No | Whether to stream back partial progress via Server-Sent Events (SSE). |
| `stop` | string[] | No | Up to 4 sequences where the model will cease generating further tokens. |
| `user` | string | No | A unique identifier representing your end-user for abuse monitoring. |
> [!NOTE]
> GoRoute's JSON parser uses an internal polymorphic unmarshaler (`CompletionPrompt`), transparently accepting either `"prompt": "Write a poem"` or `"prompt": ["First prompt", "Second prompt"]`.
### Response Schema (`CanonicalCompletionResponse`)
```json
{
"id": "cmpl-8a19b3d0-994c-4e89-9fa2-5813f02e6a12",
"model": "gpt-3.5-turbo-instruct",
"provider": "openai",
"choices": [
{
"index": 0,
"text": "\n\nConcurrency in Go is built on goroutines and channels.",
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 8,
"completion_tokens": 12,
"total_tokens": 20
}
}
```
---
## Code Examples
#### cURL
```bash
# Single string prompt
curl -X POST http://localhost:8080/v1/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo-instruct",
"prompt": "Complete the sentence: In computer science, an idempotent operation is",
"max_tokens": 64,
"temperature": 0.2
}'
# Array prompt batching
curl -X POST http://localhost:8080/v1/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo-instruct",
"prompt": [
"def fibonacci(n):",
"def quicksort(arr):"
],
"max_tokens": 128,
"temperature": 0.0
}'
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
# Single completion
completion = client.completions.create(
model="gpt-3.5-turbo-instruct",
prompt="Explain CAP theorem in one sentence:",
max_tokens=50,
temperature=0.3
)
print(completion.choices[0].text.strip())
# Streaming completion
stream = client.completions.create(
model="gpt-3.5-turbo-instruct",
prompt="Count from 1 to 5:",
max_tokens=20,
stream=True
)
for chunk in stream:
print(chunk.choices[0].text or "", end="")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type CompletionRequest struct {
Model string `json:"model"`
Prompt any `json:"prompt"`
MaxTokens int `json:"max_tokens,omitempty"`
Temperature float64 `json:"temperature,omitempty"`
Stop []string `json:"stop,omitempty"`
}
type CompletionResponse struct {
ID string `json:"id"`
Model string `json:"model"`
Choices []struct {
Index int `json:"index"`
Text string `json:"text"`
FinishReason string `json:"finish_reason"`
} `json:"choices"`
Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
} `json:"usage"`
}
func main() {
reqPayload := CompletionRequest{
Model: "gpt-3.5-turbo-instruct",
Prompt: "The primary distinction between TCP and UDP is",
MaxTokens: 60,
Temperature: 0.1,
}
data, _ := json.Marshal(reqPayload)
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/completions",
bytes.NewReader(data),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var compResp CompletionResponse
_ = json.Unmarshal(body, &compResp)
fmt.Printf("Model: %s | Tokens: %d\n", compResp.Model, compResp.Usage.TotalTokens)
if len(compResp.Choices) > 0 {
fmt.Printf("Completion: %s\n", compResp.Choices[0].Text)
}
}
```
---
## Token Quota & FinOps Ledger Integration
Requests entering `/v1/completions` follow the identical multi-layer FinOps lifecycle as chat completions:
1. **Pre-Flight Estimation**: Tokens are estimated as `PromptChars / 4` (minimum 10 tokens) and reserved via the sliding-window rate limiter.
2. **Post-Flight Reconciliation**: When upstream execution finishes, exact prompt and completion token counts from `Usage` are reconciled with the rate limiter.
3. **Micro-Dollar Ledger**: An immutable `SpendEvent` is committed asynchronously with cost calculated using 64-bit ceiling integer arithmetic:
```text
Cost (µ$) = ⌈ (PromptTokens * InputRate + CompletionTokens * OutputRate) / 1,000,000 ⌉
```
> [!TIP]
> If your application is migrating from legacy base models to instruction-tuned chat models, consider switching to [`/v1/chat/completions`](https://goroute-ai.github.io/docs/gateway/endpoints/chat-completions) for broader multi-provider routing and structured tool calling.
---
## OpenAI Responses API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/responses
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/responses.md
Section: Control Plane Documentation > Use the API
# OpenAI Responses API
GoRoute AI Gateway supports OpenAI's next-generation **Responses API** (`POST /v1/responses`). Designed as the structural successor to chat completions, the Responses API consolidates instructions, multimodal conversation context, and progressive output items under an object-oriented event paradigm.
---
## Endpoint Specification
- **HTTP Method**: `POST`
- **Supported Paths**:
- `/v1/responses`
- `/responses` (Unversioned alias)
- `/openai/v1/responses` (OpenAI wire compatibility alias)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
---
## Request & Response Schemas
### Request Schema (`CanonicalResponseRequest`)
```json
{
"model": "gpt-4.1",
"instructions": "You are a concise infrastructure assistant.",
"input": "Explain the role of an intelligent proxy in LLM gateways.",
"temperature": 0.5,
"max_output_tokens": 300,
"stream": false
}
```
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `model` | string | **Yes** | Target model (e.g. `gpt-4.1`, `fast`, `gemini/gemini-2.5-flash`). |
| `instructions`| string | No | Top-level system directives governing the assistant's behavior. |
| `input` | string \| object[] | **Yes** | User input string or array of prior conversation message objects. |
| `temperature` | float | No | Sampling temperature between `0.0` and `2.0`. |
| `max_output_tokens` | integer | No | Maximum token ceiling for generated output items. |
| `tools` | array | No | Function/tool definitions forwarded to the selected upstream provider. |
| `extra_body` | object | No | Provider-specific parameters merged without clobbering core request fields. |
| `stream` | boolean | No | When `true`, streams incremental lifecycle events via SSE. |
### Synchronous Response Schema (`CanonicalResponseOutput`)
```json
{
"id": "resp_98a7bc12-f049-411a-8c9e-28491a0123fe",
"object": "response",
"created_at": 1740000000,
"model": "gpt-4.1",
"status": "completed",
"usage": {
"prompt_tokens": 24,
"completion_tokens": 48,
"total_tokens": 72
},
"output": [
{
"id": "msg_89f1a23c_0",
"type": "message",
"status": "completed",
"role": "assistant",
"content": [
{
"type": "output_text",
"text": "An intelligent proxy acts as an abstraction layer between client applications and heterogeneous upstream model providers..."
}
]
}
]
}
```
---
## Streaming Event Lifecycle
When `"stream": true` is enabled, the Responses API delivers granular lifecycle events over Server-Sent Events (`text/event-stream`):
```http
event: response.created
data: {"response":{"id":"resp_...","object":"response","status":"in_progress","model":"gpt-4.1","created_at":1740000000}}
event: response.output_item.added
data: {"output_index":0,"item":{"id":"msg_...","type":"message","status":"in_progress","role":"assistant"}}
event: response.content_part.added
data: {"output_index":0,"content_index":0,"part":{"type":"output_text","text":""}}
event: response.text.delta
data: {"output_index":0,"content_index":0,"delta":"An intelligent"}
event: response.text.delta
data: {"output_index":0,"content_index":0,"delta":" proxy manages..."}
event: response.output_text.done
data: {"output_index":0,"content_index":0,"text":"An intelligent proxy manages..."}
event: response.output_item.done
data: {"output_index":0,"item":{"id":"msg_...","type":"message","status":"completed","role":"assistant","content":[{"type":"output_text","text":"An intelligent proxy manages..."}]}}
event: response.completed
data: {"response":{"id":"resp_...","object":"response","status":"completed","model":"gpt-4.1","created_at":1740000000,"output":[...]}}
event: response.done
data: {}
```
---
## Code Examples
#### cURL
```bash
# 1. Non-streaming call
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4.1",
"instructions": "Be strictly factual and brief.",
"input": "What is the capital of Australia?",
"stream": false
}'
# 2. SSE Streaming call via /openai/v1/responses
curl -X POST http://localhost:8080/openai/v1/responses \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4.1",
"instructions": "Count from 1 to 5.",
"input": "Begin counting.",
"stream": true
}'
```
#### Python (HTTPX)
```python
import httpx
client = httpx.Client(
base_url="http://localhost:8080",
headers={"Authorization": "Bearer sk-gw-prod-7f9a2b8c4d1e"}
)
response = client.post("/v1/responses", json={
"model": "fast",
"instructions": "You are an assistant specializing in Go concurrency.",
"input": "Describe channel buffering in 15 words or less."
})
data = response.json()
print("Status:", data["status"])
print("Output:", data["output"][0]["content"][0]["text"])
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type ResponseRequest struct {
Model string `json:"model"`
Instructions string `json:"instructions,omitempty"`
Input string `json:"input"`
Stream bool `json:"stream"`
}
type ResponseOutput struct {
ID string `json:"id"`
Status string `json:"status"`
Output []struct {
Role string `json:"role"`
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
} `json:"output"`
}
func main() {
payload, _ := json.Marshal(ResponseRequest{
Model: "gpt-4.1",
Instructions: "Answer concisely.",
Input: "What is mutual TLS?",
Stream: false,
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/responses",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var out ResponseOutput
_ = json.Unmarshal(body, &out)
if len(out.Output) > 0 && len(out.Output[0].Content) > 0 {
fmt.Printf("Response (%s): %s\n", out.ID, out.Output[0].Content[0].Text)
}
}
```
---
## Function & Tool Calling
URL: https://goroute-ai.github.io/docs/gateway/endpoints/tool-calling
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/tool-calling.md
Section: Control Plane Documentation > Use the API
# Function & Tool Calling
GoRoute AI Gateway standardizes OpenAI-format tool calling across all supported providers. It automatically translates tool definitions and execution arguments between OpenAI, Anthropic Claude, and Google Gemini, while seamlessly integrating with the **Model Context Protocol (MCP)**.
---
## Tool Definition Schema
Pass tool definitions in the `tools` array using standard JSON Schema definitions:
```json
{
"model": "fast",
"messages": [
{"role": "user", "content": "What is the stock price of AAPL and MSFT?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_stock_quote",
"description": "Returns current real-time stock price and trading volume.",
"strict": true,
"parameters": {
"type": "object",
"properties": {
"ticker": {
"type": "string",
"description": "Stock ticker symbol (e.g. AAPL, MSFT)"
},
"currency": {
"type": "string",
"enum": ["USD", "EUR", "GBP"],
"description": "Currency for quotation"
}
},
"required": ["ticker", "currency"],
"additionalProperties": false
}
}
}
],
"tool_choice": "auto"
}
```
---
## Strict Parameter Schemas (`strict: true`)
When `strict: true` is configured:
1. **Schema Constraints**: The schema must explicitly define `"additionalProperties": false` and include every property in the `"required"` array.
2. **Deterministic Output**: Downstream providers that support Structured Outputs (OpenAI, Gemini) enforce a constrained decoding grammar, guaranteeing 100% schema conformance.
3. **Gateway Pre-Flight Check**: GoRoute inspects tool parameter schemas during pre-flight admission, returning an immediate `400 Bad Request` if schema syntax is invalid.
---
## Model Context Protocol (MCP) Alignment
GoRoute natively aligns HTTP tool definitions with the Model Context Protocol:
- **Namespace Prefixing**: Tools registered on connected MCP servers can be invoked directly over standard HTTP chat completions by referencing the tool name or using the scoped format:
```
mcp____
```
- **Autonomous Execution**: If a tool call matches a registered server-side MCP tool, the gateway's [Autonomous Tool Loop](https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop) can intercept and execute it server-side, eliminating client roundtrips.
- **Bi-Directional Translation**: MCP tool schemas (`inputSchema`) map cleanly into OpenAPI parameters without manual bridging.
---
## Cross-Provider Translation Matrix
| Gateway Schema (OpenAI Format) | Anthropic Claude Wire Format | Google Gemini Wire Format |
| :--- | :--- | :--- |
| `type: "function"` | Inferred function | `functionDeclarations` |
| `function.parameters` | `input_schema` | `parameters` |
| `tool_calls[].id` | `tool_use.id` | Inferred function call call ID |
| `role: "tool"` | `role: "user"` with `type: "tool_result"` | `role: "user"` with `functionResponse` |
> [!NOTE]
> When models emit parallel tool calls, GoRoute automatically consolidates consecutive `role: "tool"` turns into a single alternating turn when routing to Anthropic or Gemini.
---
## Code Examples
#### cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "fast",
"messages": [
{"role": "user", "content": "What is the weather in Tokyo?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_weather",
"description": "Get the current weather for a given city",
"strict": true,
"parameters": {
"type": "object",
"properties": {
"city": {"type": "string"},
"unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
},
"required": ["city", "unit"],
"additionalProperties": false
}
}
}
],
"tool_choice": "auto"
}'
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
tools = [
{
"type": "function",
"function": {
"name": "calculate_tax",
"description": "Calculates sales tax based on state code.",
"strict": True,
"parameters": {
"type": "object",
"properties": {
"amount": {"type": "number"},
"state": {"type": "string"}
},
"required": ["amount", "state"],
"additionalProperties": False
}
}
}
]
response = client.chat.completions.create(
model="fast",
messages=[{"role": "user", "content": "Calculate tax for $250 in California."}],
tools=tools,
tool_choice="auto"
)
choice = response.choices[0]
if choice.message.tool_calls:
for tc in choice.message.tool_calls:
print(f"Tool to invoke: {tc.function.name}")
print(f"Arguments: {tc.function.arguments}")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
func main() {
requestBody := map[string]interface{}{
"model": "fast",
"messages": []map[string]string{
{"role": "user", "content": "Look up database metrics for cluster prod-db-1."},
},
"tools": []map[string]interface{}{
{
"type": "function",
"function": map[string]interface{}{
"name": "mcp__metrics__get_cluster_load",
"description": "Retrieves CPU and connection metrics from MCP server",
"parameters": map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"cluster_id": map[string]string{"type": "string"},
},
"required": []string{"cluster_id"},
},
},
},
},
}
payload, _ := json.Marshal(requestBody)
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/chat/completions",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
fmt.Printf("Status: %d | Body:\n%s\n", resp.StatusCode, string(body))
}
```
---
## Related Documentation
- [Server-Side Autonomous Tool Loop](https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop)
- [MCP Hub & REST Bridge](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest)
- [Go SDK MCP & A2A](https://goroute-ai.github.io/docs/sdk/mcp-and-a2a)
---
## Server-Side Autonomous Tool Loop
URL: https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop.md
Section: Control Plane Documentation > Use the API
# Server-Side Autonomous Tool Loop
GoRoute AI Gateway features a built-in server-side tool execution engine (`internal/orchestrator`). When a model responds with tool calls matching registered Model Context Protocol (MCP) servers or local functions, the orchestrator intercepts the call, executes the tool in-process, feeds the output back to the model, and recursively loops until a final synthesized answer is produced.
The client receives the final completed response in a **single HTTP turn**, eliminating network roundtrips between the client and gateway.
---
## Architectural Sequence Diagram
```mermaid
sequenceDiagram
participant Client
participant GW as GoRoute AI Gateway
participant LLM as Upstream LLM
participant MCP as MCP Tool Hub
participant Ledger as FinOps Ledger
Client->>GW: 1. POST /v1/chat/completions (Prompt + Tools)
Note over GW: Enter ToolLoop (Timeout: 60s)
GW->>LLM: 2. Forward Prompt & Tool Definitions
LLM-->>GW: 3. ToolCall: calculate_tax(amount: 250)
Note over GW: Intercept MCP ToolCall in memory
GW->>MCP: 4. Execute Tool (PerToolTimeout: 10s)
MCP-->>GW: 5. Tool Result: {tax: 20.62}
Note over GW: Append Tool Result to Message Context
GW->>LLM: 6. Feed Tool Output (Role: tool)
LLM-->>GW: 7. Synthesized: "The tax is $20.62."
Note over GW: Consolidate Multi-Turn Token Usage
GW->>Ledger: 8. Commit Atomic SpendEvent
GW-->>Client: 9. HTTP 200 OK (Final Synthesized Content)
```
---
## Safety Boundaries & Execution Budget Policy
To protect systems from infinite recursion, runaway token usage, and unresponsive tool integrations, the orchestrator enforces strict budget boundaries (`ExecutionBudgetPolicy`):
| Boundary Parameter | Default Value | Description |
| :--- | :--- | :--- |
| `MaxLoopRounds` | `5` | Maximum number of recursive LLM tool-calling roundtrips before forcing termination. |
| `CumulativeTimeout` | `60s` | Absolute wall-clock timeout for the entire multi-turn orchestration loop. |
| `PerToolTimeout` | `10s` | Maximum execution duration allocated to an individual MCP tool execution. |
| `MaxToolOutputBytes` | `512 KB` | Truncation limit on tool output strings, preventing accidental context window blowouts. |
| `MaxSpendMicros` | `0` (unlimited) | Micro-dollar financial ceiling. Aborts execution if accumulated token cost exceeds threshold. |
> [!IMPORTANT]
> If `MaxLoopRounds` is reached without a terminal completion, the gateway returns the last model response with metadata header `goroute.orchestrator/budget_exhausted: true`, enabling clients to gracefully handle incomplete agentic iterations.
---
## Anti-Double-Billing FinOps Ledger
In traditional client-orchestrated agent loops, each turn requires a distinct HTTP request (`Turn 1: user prompt`, `Turn 2: user prompt + assistant tool call + tool result`, etc.). This causes severe FinOps challenges:
- Prompt tokens for earlier turns are re-sent repeatedly and billed multiple times across disconnected HTTP records.
- Distributed network failures between client turns leave billing ledgers in desynchronized states.
### Atomic Consolidation Mechanism
GoRoute completely resolves double billing through atomic accumulation:
```go
// internal/orchestrator/tool_loop.go
accumulatedUsage.PromptTokens += resp.Usage.PromptTokens
accumulatedUsage.CompletionTokens += resp.Usage.CompletionTokens
accumulatedUsage.TotalTokens += resp.Usage.TotalTokens
```
1. **Exact-Once Ledger Commitment**: Intermediate turns do not generate separate billing transactions.
2. **Atomic SpendEvent**: Only after the orchestrator exits the loop (either upon final answer or budget exhaustion) does it commit a single consolidated `SpendEvent` to the PostgreSQL ledger.
3. **Accurate Budget Decrement**: Virtual key limits and team allocations are decremented atomically, ensuring ledger accuracy.
---
## Tool Partitioning: Server-Side vs Client-Side
When a model requests multiple tools in a single turn, the orchestrator inspects each call against the authorized MCP tool registry (`mcpToolMap`):
- **Server-Side MCP Calls**: Tools registered with the gateway or prefixed with `mcp_` / `mcp__` are executed server-side.
- **Client-Side Calls**: If any tool calls cannot be executed server-side (e.g. client UI navigation, local device APIs), the orchestrator immediately halts the loop and returns the unexecuted tool calls to the client with `finish_reason: "tool_calls"`.
### Self-Correcting JSON Recovery
If a model produces malformed JSON arguments for a server-side tool:
1. On the first failure, the orchestrator feeds the JSON parser error back to the model as a `tool` error message (`"Invalid JSON arguments for tool X: ... Please format arguments as valid JSON"`).
2. The model uses its next reasoning turn to self-correct and re-emit valid arguments.
3. If JSON parsing fails a second time, the tool is marked persistently failed, preventing infinite recovery loops.
---
## Streaming with Lookahead Buffering (`ExecuteStreamLoop`)
Streaming responses when an autonomous tool loop is active requires lookahead buffering:
- Intermediate reasoning turns and tool invocations are withheld from the client's SSE connection.
- Once the final synthesized pure-text answer is produced, the orchestrator streams the final completion tokens word-by-word or in real-time chunks, accompanied by final usage telemetry.
---
## Configuration Example (`configs/config.yaml`)
```yaml
orchestrator:
enabled: true
max_loop_rounds: 5
cumulative_timeout: 60s
per_tool_timeout: 10s
max_tool_output_bytes: 524288 # 512 KB
```
## Response Metadata Headers
Responses executed through the autonomous loop include tracking metadata:
- `goroute.orchestrator/loop_rounds`: Number of recursive turns completed (e.g. `"2"`)
- `goroute.orchestrator/total_tool_calls`: Cumulative number of tools executed (e.g. `"3"`)
- `goroute.orchestrator/budget_exhausted`: `"true"` if loop halted due to round limits
---
## Model Management & Access Control
URL: https://goroute-ai.github.io/docs/gateway/endpoints/model-management
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/model-management.md
Section: Control Plane Documentation > Use the API
# Model Management & Access Control
GoRoute AI Gateway separates physical upstream models from logical application endpoints. Through a **4-layer access control hierarchy**, organizations can govern vendor entitlements, enforce approval policies, and restrict client keys with zero-latency hot-path verification.
---
## 4-Layer Model Access Control Hierarchy
Every inference request passes through four defensive evaluation boundaries:
```mermaid
flowchart TD
Req["Incoming Request (model: 'cerebras/llama-3.3-70b')"] --> L1
subgraph L1["Layer 1: Global Model Catalogue"]
L1Check["Verify model exists, active lifecycle status,
tri-state capabilities & pricing provenance"]
end
L1 -->|Valid| L2
L1 -->|Unknown Model| Fail404["HTTP 404 Model Not Found"]
subgraph L2["Layer 2: Org Provider Subscriptions"]
L2Check["Check if Organization has enabled provider
(OrgProviderSubscription.IsEnabled = true)"]
end
L2 -->|Subscribed| L3
L2 -->|Provider Disabled| Fail403A["HTTP 403 Provider Not Subscribed"]
subgraph L3["Layer 3: Org Model Policy Overrides"]
L3Check["Check if model is approved for the org
(OrgModelPolicy.IsEnabled = true)"]
end
L3 -->|Approved| L4
L3 -->|Model Blocked| Fail403B["HTTP 403 Model Policy Blocked"]
subgraph L4["Layer 4: Virtual Key Allowlist"]
L4Check["Verify key allowed_models list
(VirtualKey.AllowedModels)"]
end
L4 -->|Authorized| Exec["Stage 2 Load Balancer & Dispatch"]
L4 -->|Key Restricted| Fail403C["HTTP 403 Key Not Authorized"]
```
### Layer Breakdown
1. **Layer 1: Global Model Catalogue (`GlobalModel`)**:
Tracks model records known to this gateway deployment, including discovered and manually registered models. Catalog contents and lifecycle status can change as provider catalogs and discovery settings change. The catalog is not a promise that every upstream model is compatible with every route or enabled for every tenant.
2. **Layer 2: Organization Provider Subscriptions (`OrgProviderSubscription`)**:
Enterprise vendor gating. An organization controls which upstream providers (`openai`, `anthropic`, `gemini`, `cohere`) are permitted for use by its developers and teams. Supports `auto_enable_new` to automatically adopt new models discovered from an approved vendor.
3. **Layer 3: Organization Model Policy Overrides (`OrgModelPolicy`)**:
Tenant model approval. Even if a provider is subscribed, security teams can explicitly approve or ban specific models (e.g. banning preview models, approving only HIPAA-compliant models). Records `approved_by` and `approved_at` audit timestamps.
4. **Layer 4: Virtual Key Allowlist (`VirtualKey.AllowedModels`)**:
Application-level restriction. Each virtual API key can define a scoped `allowed_models` array (e.g. `["fast", "text-embedding-3-small"]`). If set, callers using that key are restricted exclusively to those models.
---
## Tri-State Capability Verification
Model capabilities are notoriously prone to drift between vendor documentation and actual API support. To prevent runtime runtime errors, GoRoute replaces fragile boolean flags with an explicit **Tri-State Capability Model** (`internal/domain/capabilities.go`):
```go
type CapabilityState string
const (
CapabilitySupported CapabilityState = "supported" // Verified functional
CapabilityUnsupported CapabilityState = "unsupported" // Verified lack of support
CapabilityUnknown CapabilityState = "unknown" // Unverified / experimental
)
```
The gateway tracks 13 distinct capability dimensions per model:
| Capability Field | Description |
| :--- | :--- |
| `tools` | Function and tool calling support |
| `streaming` | Server-Sent Events (SSE) streaming tokens |
| `vision` | Multimodal image and diagram understanding |
| `reasoning` | Extended thinking / reasoning effort parameters |
| `structured_outputs`| Strict JSON Schema enforcement (`response_format`) |
| `embeddings` | Vector embedding generation |
| `rerank` | Document cross-encoder reranking |
| `moderation` | Safety and policy classification |
| `image_gen` | Text-to-image synthesis |
| `audio_stt` | Audio transcription / speech-to-text |
| `audio_tts` | Text-to-speech audio synthesis |
| `batches` | 24-hour asynchronous batch processing |
| `realtime` | Full-duplex WebSocket sessions |
> [!IMPORTANT]
> If a client requests a feature (e.g. `stream: true` or `tools`) for a model whose capability state is `unsupported`, the gateway immediately rejects the request with `HTTP 400 Bad Request` prior to upstream dispatch, preventing wasted vendor charges.
---
## Pricing Provenance & Fail-Closed Safety
To guarantee financial integrity across billing cycles, model pricing in GoRoute includes full cryptographic and audit provenance (`internal/domain/modelmgmt.go`):
- **Pricing Status**:
- `verified`: Confirmed against official provider contracts (e.g. `global-model-catalog-v1.0`).
- `custom`: Overridden by organization enterprise discount agreements.
- `unknown`: Unverified pricing tier. Triggers fail-closed budget checks.
- `unrecorded`: Free or internal local models (e.g. Ollama).
- **Pricing Provenance Source**: Explicit reference tag indicating source documentation (e.g. `"global-model-catalog-v1.0"`, `"enterprise-agreement-2026-q1"`).
- **Versioned History (`ModelPricingHistory`)**: Every rate change increments `pricing_version` and stores the previous rates with effective timestamps and changing principal.
Responses echo provenance headers:
```http
X-Gateway-Cost-Micros: 42
X-Gateway-Pricing-Status: verified
X-Gateway-Pricing-Version: 1
```
---
## Declarative Model Groups (`configs/config.yaml`)
Define load-balancing tiers, weights, and cross-provider fallbacks:
```yaml
model_list:
# Fast tier: Load-balanced between Gemini 2.5 Flash and GPT-4o-mini
- model_name: fast
routing_strategy: weighted-shuffle
model_params:
model: gemini/gemini-2.5-flash
api_key: os.environ/GEMINI_API_KEY
weight: 75
- model_name: fast
routing_strategy: weighted-shuffle
model_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
weight: 25
# Reasoning tier: High-complexity models with automatic fallback
- model_name: reasoning
routing_strategy: least-busy
model_params:
model: anthropic/claude-sonnet-5-5
api_key: os.environ/ANTHROPIC_API_KEY
fallbacks: ["backup-reasoning"]
# Local tier: Private local LLM (Ollama)
- model_name: local
model_params:
model: ollama/llama3.2:1b
api_base: http://localhost:11434
```
---
## Listing Available Models (`GET /v1/models`)
Clients can inspect configured aliases and the models permitted for their key. This tenant-filtered response is the deployment-specific check for available model IDs; it does not enumerate every model offered by an upstream provider:
```bash
curl http://localhost:8080/v1/models \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
Response:
```json
{
"object": "list",
"data": [
{"id": "fast", "object": "model", "owned_by": "goroute-gateway"},
{"id": "reasoning", "object": "model", "owned_by": "goroute-gateway"},
{"id": "auto", "object": "model", "owned_by": "goroute-gateway"},
{"id": "gemini-2.5-flash", "object": "model", "owned_by": "gemini"},
{"id": "claude-sonnet-4-20250514", "object": "model", "owned_by": "anthropic"}
]
}
```
---
## Embeddings API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/embeddings
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/embeddings.md
Section: Control Plane Documentation > Use the API
# Embeddings API
Generate dense vector embeddings across OpenAI, Google Gemini, Mistral, **Cohere Embed v3**, and **Voyage AI** using a single unified schema. Vector embeddings transform text into high-dimensional numerical vectors for semantic search, retrieval-augmented generation (RAG), clustering, and classification.
---
## Endpoint Specification
- **Method**: `POST`
- **Path**: `/v1/embeddings` (also supports unversioned `/embeddings`)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
---
## Supported Providers & Model Families
GoRoute standardizes embedding generation across leading model providers:
| Provider | Notable Models | Dimensions | Features |
| :--- | :--- | :--- | :--- |
| **OpenAI** | `text-embedding-3-small`, `text-embedding-3-large` | 1536 / 3072 | Matryoshka dimension truncation, float & base64. |
| **Cohere** | `cohere/embed-english-v3.0`, `cohere/embed-multilingual-v3.0` | 1024 | Task-specific optimization (`search_document`, `search_query`). |
| **Voyage AI** | `voyage/voyage-3`, `voyage/voyage-3-large`, `voyage/voyage-code-3` | 1024 / 1536 | Optimized for code search, legal, and enterprise RAG. |
| **Google** | `gemini/text-embedding-004` | 768 | High-throughput low-latency multilingual embeddings. |
| **Mistral** | `mistral/mistral-embed` | 1024 | European data locality and multi-language support. |
---
## Request & Response Schemas
### Request Schema (`CanonicalEmbeddingRequest`)
```json
{
"model": "text-embedding-3-small",
"input": [
"GoRoute AI Gateway provides sub-millisecond routing.",
"Pure Go compiles to native machine code without Cgo."
],
"encoding_format": "float",
"dimensions": 512
}
```
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `model` | string | **Yes** | Model identifier or provider alias (e.g. `text-embedding-3-small`, `voyage/voyage-3`). |
| `input` | string \| string[] | **Yes** | Input text or array of input texts to vectorize. |
| `encoding_format` | string | No | `float` (default) or `base64`. Base64 yields smaller HTTP payload sizes. |
| `dimensions` | integer | No | Truncate dimensions on models supporting Matryoshka embeddings (e.g. OpenAI v3). |
| `user` | string | No | Optional end-user identifier for abuse monitoring. |
### Response Schema (`CanonicalEmbeddingResponse`)
```json
{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.0123, -0.0456, 0.0789, "..."]
}
],
"model": "text-embedding-3-small",
"usage": {
"prompt_tokens": 16,
"total_tokens": 16
}
}
```
> [!NOTE]
> When `encoding_format: "base64"` is requested, each item's `embedding` contains a base64-encoded string representing the Little-Endian IEEE 754 binary floating-point array, dramatically shrinking response payload size over HTTP.
---
## Code Examples
#### cURL
```bash
# 1. OpenAI Embeddings with Matryoshka dimension reduction
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "text-embedding-3-small",
"input": [
"GoRoute AI Gateway provides sub-millisecond routing.",
"Pure Go compiles to native machine code without Cgo."
],
"dimensions": 512
}'
# 2. Voyage AI Code Embeddings
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "voyage/voyage-code-3",
"input": "func ServeHTTP(w http.ResponseWriter, r *http.Request)"
}'
# 3. Cohere Embed v3
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere/embed-english-v3.0",
"input": ["Document text to index in vector database"]
}'
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
# Standard embeddings with dimension customization
response = client.embeddings.create(
model="text-embedding-3-large",
input=["Vector search with GoRoute AI Gateway", "Distributed embeddings cache"],
dimensions=1024
)
for item in response.data:
print(f"Index {item.index}: Vector length = {len(item.embedding)}")
# Voyage AI model via same client
voyage_resp = client.embeddings.create(
model="voyage/voyage-3",
input=["Retrieval augmented generation in finance"]
)
print("Voyage embedding dimension:", len(voyage_resp.data[0].embedding))
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type EmbeddingRequest struct {
Model string `json:"model"`
Input []string `json:"input"`
Dimensions int `json:"dimensions,omitempty"`
}
type EmbeddingResponse struct {
Model string `json:"model"`
Data []struct {
Index int `json:"index"`
Embedding []float64 `json:"embedding"`
} `json:"data"`
Usage struct {
TotalTokens int `json:"total_tokens"`
} `json:"usage"`
}
func main() {
payload, _ := json.Marshal(EmbeddingRequest{
Model: "text-embedding-3-small",
Input: []string{"Zero allocation networking in Go."},
Dimensions: 256,
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/embeddings",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var embResp EmbeddingResponse
_ = json.Unmarshal(body, &embResp)
fmt.Printf("Model: %s | Total Tokens: %d\n", embResp.Model, embResp.Usage.TotalTokens)
if len(embResp.Data) > 0 {
fmt.Printf("Vector dimension: %d\n", len(embResp.Data[0].Embedding))
}
}
```
---
## Response Telemetry & FinOps Ledger
Every embeddings request returns:
- `X-Gateway-Request-Id`: Unique request UUID (`req_...`)
- `X-Gateway-Provider`: Provider used (`openai`, `voyage`, `cohere`)
- `X-Gateway-Cost-Micros`: Exact micro-dollar cost based on input tokens
- `X-Gateway-Pricing-Status`: `verified`, `custom`, or `unknown`
> [!TIP]
> Use [`POST /v1/tokenize`](https://goroute-ai.github.io/docs/gateway/endpoints/tokenize) to pre-calculate embedding batch sizes and stay within upstream token limits without running extra network calls.
---
## Document Reranking API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/rerank
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/rerank.md
Section: Control Plane Documentation > Use the API
# Document Reranking API
The GoRoute Data Plane provides a unified semantic reranking API compatible with **Cohere v2** (`/v2/rerank`) and **Jina AI**. Reranking takes an initial set of candidate documents retrieved via lexical or vector search and reorders them using cross-encoder deep neural models for optimal relevance against a search query.
---
## Endpoint Specification
- **HTTP Methods**: `POST`
- **Supported Paths**:
- `/v1/rerank` (Standardized gateway endpoint)
- `/rerank` (Unversioned compatibility alias)
- `/v2/rerank` (Cohere v2 wire specification alias)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
---
## Request & Response Schemas
### Request Schema (`CanonicalRerankRequest`)
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `model` | string | **Yes** | Target reranker model or provider/model (e.g. `cohere/rerank-v3.5`, `jina/jina-reranker-v2-base-multilingual`). |
| `query` | string | **Yes** | The search query against which candidate documents are scored. |
| `documents` | array | **Yes** | Candidate documents. Accepts either an array of plain strings (`string[]`) or an array of arbitrary JSON objects (`object[]`). |
| `top_n` | integer | No | Maximum number of top-ranked results to return. If omitted, all documents are returned in ranked order. |
| `rank_fields` | string[] | No | When `documents` contains structured objects, specifies which JSON keys to consider for relevance ranking (e.g. `["title", "content"]`). |
| `return_documents` | boolean | No | Whether to return the document payload inside each result item. Default: `false`. |
| `max_tokens_per_doc` | integer | No | Maximum tokens per document to evaluate. Documents exceeding this length will be truncated by the upstream cross-encoder. |
### Response Schema (`CanonicalRerankResponse`)
```json
{
"id": "rerank_c7b5f90a-1123-4f9e-8c3b-280145c2ee01",
"model": "rerank-v3.5",
"provider": "cohere",
"results": [
{
"index": 1,
"relevance_score": 0.9842,
"document": {
"text": "GoRoute AI Gateway is built with pure Go 1.24 and avoids Cgo overhead."
}
},
{
"index": 0,
"relevance_score": 0.3129,
"document": {
"text": "The solar system contains eight major planets orbiting the Sun."
}
}
],
"meta": {
"api_version": {
"version": "2"
},
"billed_units": {
"search_units": 1
}
},
"usage": {
"search_units": 1,
"total_tokens": 84
}
}
```
> [!NOTE]
> The `index` field in each result item corresponds to the original 0-based position of the document in the input `documents` array, allowing zero-copy mapping back to your local candidate cache.
---
## Code Examples
#### cURL
```bash
# Plain string documents
curl -X POST http://localhost:8080/v1/rerank \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere/rerank-v3.5",
"query": "high performance zero allocation proxy",
"documents": [
"Traditional Python proxies incur GIL locking and GC latency pauses.",
"GoRoute AI Gateway achieves sub-millisecond p99 latency with pure Go zero-allocation buffers.",
"PostgreSQL stores virtual key credentials with AES-256-GCM envelope encryption."
],
"top_n": 2,
"return_documents": true
}'
# Structured object documents with rank_fields via /v2/rerank alias
curl -X POST http://localhost:8080/v2/rerank \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "jina/jina-reranker-v2-base-multilingual",
"query": "how to configure upstream fallbacks",
"documents": [
{
"title": "Reliability & Fallbacks",
"section": "configuration",
"content": "Configure fallback_group_ids in configs/config.yaml for seamless provider failover."
},
{
"title": "Audio Speech Synthesis",
"section": "audio",
"content": "Speech synthesis maps characters using integer ceiling division to micro-dollars."
}
],
"rank_fields": ["title", "content"],
"top_n": 1,
"return_documents": true
}'
```
#### Python
```python
import httpx
client = httpx.Client(
base_url="http://localhost:8080",
headers={"Authorization": "Bearer sk-gw-prod-7f9a2b8c4d1e"}
)
response = client.post("/v1/rerank", json={
"model": "cohere/rerank-v3.5",
"query": "zero allocation memory buffers",
"documents": [
"GoRoute streaming uses zero-allocation sync.Pool buffers.",
"Node.js event loops can block on large string concatenations.",
"Redis provides distributed sliding window rate limiting."
],
"top_n": 2,
"return_documents": True
})
data = response.json()
print(f"Provider: {response.headers.get('X-Gateway-Provider')}")
print(f"Cost: {response.headers.get('X-Gateway-Cost-Micros')} micro-dollars")
for item in data["results"]:
print(f"Rank score: {item['relevance_score']:.4f} (Original index: {item['index']})")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type RerankRequest struct {
Model string `json:"model"`
Query string `json:"query"`
Documents []string `json:"documents"`
TopN int `json:"top_n,omitempty"`
ReturnDocuments bool `json:"return_documents"`
}
type RerankResult struct {
Index int `json:"index"`
RelevanceScore float64 `json:"relevance_score"`
Document map[string]interface{} `json:"document,omitempty"`
}
type RerankResponse struct {
ID string `json:"id"`
Model string `json:"model"`
Provider string `json:"provider"`
Results []RerankResult `json:"results"`
Usage struct {
SearchUnits int `json:"search_units"`
TotalTokens int `json:"total_tokens"`
} `json:"usage"`
}
func main() {
reqBody := RerankRequest{
Model: "cohere/rerank-v3.5",
Query: "sub-millisecond gateway latency",
Documents: []string{
"High latency Python backends suffer from interpreter overhead.",
"GoRoute provides sub-millisecond p99 latency using pure Go and Chi router.",
"Docusaurus produces static developer documentation portals.",
},
TopN: 2,
ReturnDocuments: true,
}
payload, _ := json.Marshal(reqBody)
req, _ := http.NewRequestWithContext(context.Background(), http.MethodPost, "http://localhost:8080/v1/rerank", bytes.NewReader(payload))
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var rerankResp RerankResponse
_ = json.Unmarshal(body, &rerankResp)
fmt.Printf("Status: %d | Provider: %s | Search Units: %d\n",
resp.StatusCode, rerankResp.Provider, rerankResp.Usage.SearchUnits)
for _, r := range rerankResp.Results {
fmt.Printf(" [Score: %.4f] Index %d: %v\n", r.RelevanceScore, r.Index, r.Document["text"])
}
}
```
---
## FinOps Accounting & Micro-Dollar Ledger
Rerank workloads do not consume standard prompt/completion tokens; upstream providers (like Cohere) bill based on **Search Units** (where 1 Search Unit typically represents up to 100 candidate documents processed in a query).
GoRoute AI Gateway enforces micro-dollar integer accounting:
```text
Cost (µ$) = search_units * RerankCostMicrosPerSearchUnit
```
> [!IMPORTANT]
> If a provider catalog record does not have a verified `RerankCostMicrosPerSearchUnit`, the gateway tags the request with `X-Gateway-Pricing-Status: unknown` and falls back defensively, preventing ledger divergence.
### Telemetry Headers
Each rerank response carries telemetry headers for observability:
- `X-Gateway-Request-Id`: Unique tracking ID (`req_...`)
- `X-Gateway-Provider`: Winning upstream provider (`cohere` or `jina`)
- `X-Gateway-Cost-Micros`: Exact micro-dollar charge deducted from the virtual key's budget
- `X-Gateway-Pricing-Status`: `verified`, `custom`, or `unknown`
---
## Audio Transcription, Translation & Speech API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/audio
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/audio.md
Section: Control Plane Documentation > Use the API
# Audio Transcription, Translation & Speech API
GoRoute AI Gateway provides full OpenAI-compatible support for multi-modal audio processing across three core capabilities:
1. **Speech-to-Text Transcription** (`POST /v1/audio/transcriptions`): Transcribes audio in its original language using OpenAI Whisper or open-source equivalents.
2. **Audio Translation** (`POST /v1/audio/translations`): Transcribes audio from any supported source language directly into English.
3. **Text-to-Speech Synthesis** (`POST /v1/audio/speech`): Synthesizes lifelike spoken audio from text using multi-voice neural models (`tts-1`, `tts-1-hd`).
---
## Endpoint Specifications
| Endpoint | Method | Input Type | Description |
| :--- | :--- | :--- | :--- |
| `/v1/audio/transcriptions` | `POST` | `multipart/form-data` | Converts spoken audio into text. |
| `/v1/audio/translations` | `POST` | `multipart/form-data` | Transcribes and translates non-English audio to English. |
| `/v1/audio/speech` | `POST` | `application/json` | Generates streaming audio from textual input. |
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Supported Audio Codecs**: `mp3`, `opus`, `aac`, `flac`, `wav`, `m4a`, `webm`
- **Max Audio File Upload**: `25 MB` (enforced via `http.MaxBytesReader`)
---
## Architecture: Zero-Memory Retryable Disk Spooling
When handling large audio uploads, loading entire audio files into memory buffers would cause severe memory pressure and garbage collection pauses under high concurrent loads. Furthermore, if an upstream provider fails mid-flight, a pure streaming pipeline cannot retry because the HTTP request body stream has already been consumed.
GoRoute solves this with **Zero-Memory Disk Spooling**:
```mermaid
sequenceDiagram
participant Client
participant GW as GoRoute AI Gateway
participant ProvA as Primary Provider (STT)
participant ProvB as Fallback Provider (STT)
Client->>GW: 1. POST /v1/audio/transcriptions (Multipart 25MB)
GW->>GW: 2. Spool stream to disk (gw-audio-*.tmp)
GW->>ProvA: 3. Stream from disk spool
ProvA-->>GW: 4. HTTP 503 Service Unavailable
GW->>GW: 5. Rewind file descriptor (Seek 0, Zero GC)
GW->>ProvB: 6. Re-stream from disk spool
ProvB-->>GW: 7. HTTP 200 OK (Transcription JSON)
GW->>GW: 8. Unlink & remove spool file
GW-->>Client: 9. Return Transcript + Telemetry
```
1. **Disk Spooling**: Incoming multipart files are written directly to a temporary file (`gw-audio-*.tmp`) in chunks using `io.Copy`.
2. **Infinite Rewindability**: The gateway's reliability layer can rewind the file descriptor (`Seek(0, io.SeekStart)`) to replay the exact audio bytes to healthy fallback deployments without re-reading from the client.
3. **Immediate Cleanup**: Once the upstream completes (or fails all retries), the temporary file is safely unlinked (`defer os.Remove(spoolPath)`).
---
## Speech-to-Text Transcription (`/v1/audio/transcriptions`)
### Multipart Parameters
- `file` (binary, required): The audio file to transcribe.
- `model` (string, optional, default: `whisper-1`): Speech model identifier.
- `language` (string, optional): ISO-639-1 language code (e.g. `en`, `es`, `fr`, `de`).
- `prompt` (string, optional): An optional text guide to establish spelling, jargon, or punctuation context.
- `response_format` (string, optional, default: `json`): Format of the transcript: `json`, `text`, `verbose_json`, `srt`, `vtt`.
- `temperature` (float, optional, default: `0`): Sampling temperature between `0` and `1`.
### FinOps Accounting for STT
Whisper models bill based on **audio duration** rather than text tokens. GoRoute records exact duration in integer milliseconds (`duration_ms`) and computes spend in micro-dollars using 64-bit integer ceiling arithmetic:
```text
Cost (µ$) = ⌈ (DurationMs * AudioSTTCostMicrosPerSecond + 999) / 1000 ⌉
```
---
## Text-to-Speech Synthesis (`/v1/audio/speech`)
### JSON Request Schema
```json
{
"model": "tts-1",
"input": "GoRoute AI Gateway synthesizes audio with low latency and streaming responses.",
"voice": "alloy",
"response_format": "mp3",
"speed": 1.0
}
```
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `model` | string | No | TTS model (`tts-1` or `tts-1-hd`). Default: `tts-1`. |
| `input` | string | **Yes** | Text to convert to speech (up to 4096 characters). |
| `voice` | string | No | Voice profile: `alloy`, `echo`, `fable`, `onyx`, `nova`, `shimmer`. Default: `alloy`. |
| `response_format` | string | No | Output format: `mp3`, `opus`, `aac`, `flac`, `wav`, `pcm`. Default: `mp3`. |
| `speed` | float | No | Playback speed from `0.25` to `4.0`. Default: `1.0`. |
### FinOps Accounting for TTS
TTS is priced strictly per 1,000 input characters. GoRoute enforces ceiling integer division so that short sentences never truncate fractional micro-dollars to zero:
```text
Cost (µ$) = ⌈ (CharCount * AudioTTSCostMicrosPer1KChars + 999) / 1000 ⌉
```
---
## Code Examples
#### cURL
```bash
# 1. Transcribe Audio
curl -X POST http://localhost:8080/v1/audio/transcriptions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@meeting_recording.mp3" \
-F "model=whisper-1" \
-F "language=en"
# 2. Translate Foreign Audio to English
curl -X POST http://localhost:8080/v1/audio/translations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@spanish_interview.wav" \
-F "model=whisper-1"
# 3. Text-to-Speech Synthesis
curl -X POST http://localhost:8080/v1/audio/speech \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "tts-1",
"input": "Welcome to GoRoute AI Gateway. High reliability multi-modal routing.",
"voice": "nova",
"response_format": "mp3"
}' \
--output welcome.mp3
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
# Transcribe
with open("lecture.mp3", "rb") as audio_file:
transcript = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
response_format="text"
)
print("Transcript:", transcript)
# Synthesize Speech
speech_response = client.audio.speech.create(
model="tts-1",
voice="alloy",
input="Streaming real-time TTS over GoRoute Gateway."
)
speech_response.stream_to_file("output.mp3")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"time"
)
type SpeechRequest struct {
Model string `json:"model"`
Input string `json:"input"`
Voice string `json:"voice"`
ResponseFormat string `json:"response_format"`
}
func main() {
payload, _ := json.Marshal(SpeechRequest{
Model: "tts-1",
Input: "Zero-memory disk spooling and precision FinOps.",
Voice: "echo",
ResponseFormat: "mp3",
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/audio/speech",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
costHeader := resp.Header.Get("X-Gateway-Cost-Micros")
providerHeader := resp.Header.Get("X-Gateway-Provider")
fmt.Printf("TTS Provider: %s | Cost: %s µ$\n", providerHeader, costHeader)
outFile, _ := os.Create("synthesized.mp3")
defer outFile.Close()
_, _ = io.Copy(outFile, resp.Body)
fmt.Println("Audio saved to synthesized.mp3")
}
```
---
## Response Headers & Telemetry
| Header | Description |
| :--- | :--- |
| `X-Gateway-Request-Id` | Unique ID prefixed with `stt_`, `stt_translation_`, or `tts_` |
| `X-Gateway-Provider` | Upstream provider executing audio inference (e.g. `openai`) |
| `X-Gateway-Cost-Micros` | Micro-dollar spend incurred by the operation |
| `X-Gateway-Pricing-Status` | Price provenance (`verified`, `custom`, `unknown`) |
---
## Image Generation, Edits & Variations API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/images
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/images.md
Section: Control Plane Documentation > Use the API
# Image Generation, Edits & Variations API
GoRoute AI Gateway provides multi-provider image generation, inpainting/editing, and variation capabilities compatible with OpenAI's Images specification. Traffic can be directed to OpenAI **DALL-E 2 / DALL-E 3** or **Google Imagen** using unified request parameters.
---
## Endpoint Specifications
| Endpoint | Method | Format | Description |
| :--- | :--- | :--- | :--- |
| `/v1/images/generations` | `POST` | `application/json` | Generates new images from text prompts (DALL-E 3, Imagen). |
| `/v1/images/edits` | `POST` | `multipart/form-data` | Inpainting / editing existing images with prompt and optional alpha mask. |
| `/v1/images/variations` | `POST` | `multipart/form-data` | Creates visual variations of an uploaded source image. |
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Max Image Upload Limit**: `20 MB` (enforced via `http.MaxBytesReader`)
- **Supported Formats**: PNG (must contain standard PNG signature `\x89PNG\r\n\x1a\n`)
---
## Resolution Pricing Matrix & FinOps Accounting
Image models do not bill on input/output token counts; instead, they bill per generated image unit based on **Resolution** (`1024x1024`, `1024x1792`, `1792x1024`), **Quality Tier** (`standard` vs `hd`), and **Quantity** (`n`).
GoRoute evaluates image spend using the model catalog's `ModalityPricingJSON` and integer micro-dollar calculation:
```text
Cost (µ$) = UnitRate(Quality, Size) * n
```
### DALL-E 3 Pricing Matrix Reference
| Resolution | Quality Tier | Unit Cost (USD) | Micro-Dollars ($\mu\$) |
| :--- | :--- | :--- | :--- |
| `1024x1024` | `standard` | $0.040 | **40,000 µ$** |
| `1024x1024` | `hd` | $0.080 | **80,000 µ$** |
| `1024x1792` / `1792x1024` | `standard` | $0.080 | **80,000 µ$** |
| `1024x1792` / `1792x1024` | `hd` | $0.120 | **120,000 µ$** |
> [!NOTE]
> If a request specifies `quality: "hd"` and the model catalog lacks an explicit resolution entry, GoRoute falls back safely to `ImageCostMicrosHD`, ensuring continuous FinOps ledger tracking.
---
## Generations API (`POST /v1/images/generations`)
### Request Schema (`CanonicalImageRequest`)
```json
{
"model": "dall-e-3",
"prompt": "A modern data center rendered with glowing fiber optic cables and neon illumination.",
"n": 1,
"size": "1024x1024",
"quality": "standard",
"response_format": "url",
"style": "vivid"
}
```
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `prompt` | string | **Yes** | Text description of the desired image(s). |
| `model` | string | No | Model identifier (`dall-e-3`, `dall-e-2`, `imagen-3.0`). Default: `dall-e-3`. |
| `n` | integer | No | Number of images to generate (1 for DALL-E 3; 1–10 for DALL-E 2). Default: `1`. |
| `quality` | string | No | Image fidelity: `standard` or `hd`. Default: `standard`. |
| `size` | string | No | Resolution: `1024x1024`, `1024x1792`, `1792x1024`, `512x512`, `256x256`. |
| `response_format` | string | No | Format of returned images: `url` or `b64_json`. Default: `url`. |
| `style` | string | No | `vivid` (hyper-real/dramatic) or `natural`. Default: `vivid`. |
### Response Schema (`CanonicalImageResponse`)
```json
{
"created": 1740000000,
"model": "dall-e-3",
"provider": "openai",
"data": [
{
"url": "https://oaidalleapiprodscus.blob.core.windows.net/private/org-...",
"revised_prompt": "A sprawling server room bathed in electric blue and neon purple light..."
}
]
}
```
---
## Image Edits & Variations
### Inpainting / Edits (`POST /v1/images/edits`)
Image editing modifies an existing PNG image using a replacement prompt and an optional alpha transparency mask.
- `image`: Original PNG file (under 20 MB).
- `mask` (optional): Transparent PNG where the transparent area denotes what should be repainted.
- `prompt`: Textual instruction for the edit.
### Variations (`POST /v1/images/variations`)
Generates stylistic variants of an existing PNG image without prompt text.
- `image`: Original source PNG file.
- `n`: Number of variations to create.
---
## Code Examples
#### cURL
```bash
# 1. Generate an image
curl -X POST http://localhost:8080/v1/images/generations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "dall-e-3",
"prompt": "Minimalist geometric logo representing network routing, vector style.",
"size": "1024x1024",
"quality": "standard"
}'
# 2. Inpaint an existing image
curl -X POST http://localhost:8080/v1/images/edits \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "image=@diagram.png" \
-F "mask=@mask.png" \
-F "prompt=Add a glowing green database cluster icon inside the masked area" \
-F "size=1024x1024"
# 3. Create image variations
curl -X POST http://localhost:8080/v1/images/variations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "image=@logo.png" \
-F "n=2" \
-F "size=1024x1024"
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
response = client.images.generate(
model="dall-e-3",
prompt="Futuristic software architecture blueprint on dark glass, isometric 3D.",
size="1024x1024",
quality="hd",
n=1
)
image_url = response.data[0].url
print(f"Generated Image URL: {image_url}")
print(f"Revised Prompt: {response.data[0].revised_prompt}")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type ImageGenRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
Size string `json:"size"`
Quality string `json:"quality"`
}
type ImageGenResponse struct {
Created int64 `json:"created"`
Model string `json:"model"`
Data []struct {
URL string `json:"url"`
RevisedPrompt string `json:"revised_prompt"`
} `json:"data"`
}
func main() {
payload, _ := json.Marshal(ImageGenRequest{
Model: "dall-e-3",
Prompt: "Cyberpunk cityscape at sunset with flying vehicles.",
Size: "1024x1024",
Quality: "standard",
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/images/generations",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 60 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var imgResp ImageGenResponse
_ = json.Unmarshal(body, &imgResp)
fmt.Printf("Provider: %s | Cost: %s µ$\n",
resp.Header.Get("X-Gateway-Provider"),
resp.Header.Get("X-Gateway-Cost-Micros"),
)
if len(imgResp.Data) > 0 {
fmt.Printf("Image URL: %s\n", imgResp.Data[0].URL)
}
}
```
---
## OpenAI Realtime WebSocket Protocol
URL: https://goroute-ai.github.io/docs/gateway/endpoints/realtime
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/realtime.md
Section: Control Plane Documentation > Use the API
# OpenAI Realtime WebSocket Protocol
GoRoute AI Gateway supports the **OpenAI Realtime API** protocol over bidirectional WebSockets. Realtime enables low-latency, full-duplex conversational sessions supporting streaming text and audio with sub-second voice interactions.
---
## Endpoint Specification
- **HTTP Method**: `GET` (WebSocket Upgrade Handshake)
- **Supported Paths**:
- `/v1/realtime` (Standard data plane endpoint)
- `/realtime` (Unversioned compatibility alias)
- `/openai/v1/realtime` (OpenAI wire compatibility alias)
- **Subprotocol**: `realtime` (passed via `Sec-WebSocket-Protocol: realtime`)
- **Authentication**: `Bearer sk-gw-...` passed in the `Authorization` header during the WebSocket upgrade handshake
- **Default Model**: `gpt-4o-realtime-preview` (configurable via `?model=...` query parameter)
---
## Architecture: Zero-Allocation Connection Hijacking
GoRoute utilizes [`nhooyr.io/websocket`](https://github.com/nhooyr/websocket) for RFC 6455 compliance without external daemon processes. When a client initiates the handshake:
```mermaid
sequenceDiagram
participant Client
participant GW as GoRoute AI Gateway
participant LLM as Upstream Realtime Engine
Client->>GW: 1. GET /v1/realtime (WebSocket Upgrade)
Note over GW: Authenticate Key & Model RBAC
GW->>Client: 2. HTTP 101 Switching Protocols
Note over GW: Hijack TCP Connection
GW-->>Client: 3. ServerEvent: session.created
loop Full-Duplex Stream Loop
Client->>GW: 4. ClientEvent: response.create
GW->>LLM: 5. Stream Realtime Audio Frames
LLM-->>GW: 6. Raw Tokens / Audio Deltas
GW-->>Client: 7. ServerEvent: audio.delta
end
Client->>GW: 8. WebSocket Close
Note over GW: Enqueue SpendEvent to FinOps Ledger
```
1. **In-Flight Authentication**: Handshake requests pass through `auth.DataPlaneMiddleware`, verifying the virtual key and validating model access permissions before the TCP connection is hijacked.
2. **Session State Isolation**: Each active WebSocket maintains its own isolated message history, active tools, and audio buffer frames.
3. **Session FinOps Ledger**: When the session closes, the gateway aggregates all tokens and audio duration generated throughout the session and records a single consolidated spend transaction.
---
## Realtime Event Protocol
Communication occurs via JSON-formatted frames over text message blocks.
### 1. Client Events (Inbound)
| Event Type | Description |
| :--- | :--- |
| `session.update` | Modifies active session parameters (modalities, voice, temperature, instructions). |
| `conversation.item.create` | Appends a new user message or tool result to conversation history. |
| `input_audio_buffer.append` | Appends base64-encoded PCM16 audio bytes to the input buffer. |
| `input_audio_buffer.commit` | Commits pending audio and triggers automatic turn detection. |
| `response.create` | Instructs the model to generate an assistant turn immediately. |
| `response.cancel` | Interrupts current streaming generation (e.g. user barge-in). |
### 2. Server Events (Outbound)
| Event Type | Description |
| :--- | :--- |
| `session.created` | Emitted immediately upon connection with active session configuration. |
| `session.updated` | Acknowledges receipt and application of `session.update`. |
| `conversation.item.created` | Confirms item addition to session context. |
| `response.created` | Signals that a new response generation run has begun. |
| `response.text.delta` | Incremental text token delta from the assistant. |
| `response.audio.delta` | Incremental base64-encoded PCM16 audio chunks. |
| `response.done` | Signals completion of the response turn with final usage metadata. |
| `error` | Emitted if an invalid event, schema mismatch, or budget error occurs. |
---
## Session Configuration (`session.created`)
Immediately upon upgrading the connection, GoRoute dispatches the initial session configuration:
```json
{
"event_id": "event_88fa2b10-2180-4ce9-9f01-38290fa1b203",
"type": "session.created",
"session": {
"id": "sess_1189ac02-990a-412e-b998-382910fa8821",
"object": "realtime.session",
"model": "gpt-4o-realtime-preview",
"modalities": ["text"],
"instructions": "You are a helpful AI assistant.",
"voice": "alloy",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16"
}
}
```
---
## Code Examples
#### Python (WebSockets)
```python
import asyncio
import json
import websockets
GATEWAY_WS_URL = "ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview"
API_KEY = "sk-gw-prod-7f9a2b8c4d1e"
async def realtime_chat():
headers = {"Authorization": f"Bearer {API_KEY}"}
async with websockets.connect(GATEWAY_WS_URL, extra_headers=headers, subprotocols=["realtime"]) as ws:
# 1. Wait for session.created
init_event = json.loads(await ws.recv())
print(f"Session established: {init_event['session']['id']}")
# 2. Send user text message
await ws.send(json.dumps({
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "What is connection hijacking in HTTP servers?"}]
}
}))
# 3. Request response generation
await ws.send(json.dumps({"type": "response.create"}))
# 4. Stream response events
while True:
msg = json.loads(await ws.recv())
event_type = msg.get("type")
if event_type == "response.text.delta":
print(msg.get("delta", ""), end="", flush=True)
elif event_type == "response.done":
print("\n[Turn completed]")
break
asyncio.run(realtime_chat())
```
#### TypeScript / Browser
```typescript
const ws = new WebSocket(
"ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview",
["realtime"]
);
ws.onopen = () => {
console.log("WebSocket connected to GoRoute Gateway");
};
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
switch (data.type) {
case "session.created":
console.log("Session ID:", data.session.id);
// Send a user prompt
ws.send(
JSON.stringify({
type: "conversation.item.create",
item: {
type: "message",
role: "user",
content: [{ type: "input_text", text: "Explain WebSockets vs SSE." }],
},
})
);
ws.send(JSON.stringify({ type: "response.create" }));
break;
case "response.text.delta":
process.stdout.write(data.delta);
break;
case "response.done":
console.log("\nFinished turn.");
ws.close();
break;
}
};
```
#### Go
```go
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"time"
"nhooyr.io/websocket"
)
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
opts := &websocket.DialOptions{
Subprotocols: []string{"realtime"},
HTTPHeader: http.Header{
"Authorization": []string{"Bearer sk-gw-prod-7f9a2b8c4d1e"},
},
}
conn, _, err := websocket.Dial(ctx, "ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview", opts)
if err != nil {
panic(err)
}
defer conn.Close(websocket.StatusNormalClosure, "done")
// Read initial session.created
_, data, err := conn.Read(ctx)
if err != nil {
panic(err)
}
var sessEv map[string]interface{}
_ = json.Unmarshal(data, &sessEv)
fmt.Printf("Connected: Event %s\n", sessEv["type"])
}
```
---
## Stateful Tenant Files API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/files
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/files.md
Section: Control Plane Documentation > Use the API
# Stateful Tenant Files API
GoRoute AI Gateway implements OpenAI-compatible stateful file management to support asynchronous batch processing (`/v1/batches`), fine-tuning workflows, and multi-modal assistants. Files are isolated per organization, validated for integrity via streaming cryptographic hashes, and stored with strict POSIX permissions.
---
## Endpoint Specification
| Endpoint | Method | Output | Description |
| :--- | :--- | :--- | :--- |
| `/v1/files` | `POST` | `CanonicalFile` JSON | Upload a new tenant file (up to 512 MB). |
| `/v1/files` | `GET` | List JSON | List all active files for the authenticated organization. |
| `/v1/files/{file_id}` | `GET` | `CanonicalFile` JSON | Retrieve metadata for a specific file. |
| `/v1/files/{file_id}/content` | `GET` | Binary Stream | Download raw binary or text file contents. |
| `/v1/files/{file_id}` | `DELETE`| Deletion Status | Permanently delete file from database and disk storage. |
- **Authentication**: `Bearer sk-gw-...` (Virtual Key identity required)
- **Max Upload Limit**: `512 MB` per file (enforced via `http.MaxBytesReader`)
- **Per-Organization Storage Quota**: `5 GiB` default (`server.max_file_storage_bytes_per_org` / `GATEWAY_MAX_FILE_STORAGE_BYTES_PER_ORG`; returns `HTTP 413 storage_quota_exceeded` when exceeded)
- **Upload Concurrency Limit**: `8` concurrent uploads default (`server.max_concurrent_file_uploads` / `GATEWAY_MAX_CONCURRENT_FILE_UPLOADS`; returns `HTTP 429 upload_concurrency_limit` when saturated)
- **Default Storage Path**: Configured via `GOROUTE_FILES_DIR` or defaults to `/tmp/goroute-files`
---
## Architecture: Multi-Tenant Storage & Streaming Hashes
```mermaid
sequenceDiagram
participant Client
participant GW as GoRoute AI Gateway
participant Disk as Local Disk (0600 mode)
participant Hasher as SHA-256 Digest
participant DB as PostgreSQL
Client->>GW: 1. POST /v1/files (multipart/form-data)
Note over GW: Auth & Tenant Isolation Check
GW->>Disk: 2. Stream chunk to file descriptor
GW->>Hasher: 3. io.MultiWriter feeds SHA-256
Disk-->>GW: 4. Sync file to disk (0700 dir)
GW->>DB: 5. INSERT INTO files (id, org_id, sha256)
DB-->>GW: 6. Commit OK
GW-->>Client: 7. HTTP 201 Created (File JSON)
```
### Security & Integrity Controls
1. **POSIX File System Isolation**:
The gateway daemon creates its storage root with directory permissions `0700` (`drwx------`) and all uploaded blobs with file permissions `0600` (`-rw-------`). Unprivileged local system processes cannot read or modify tenant files.
2. **Streaming Cryptographic Hashing**:
Rather than buffering up to 512 MB in memory to calculate hashes, GoRoute utilizes an `io.MultiWriter(dst, hasher)` that writes to disk while concurrently streaming bytes into a SHA-256 digest engine.
3. **Multi-Tenant Boundary & Quotas**:
Every database query filters by `WHERE org_id = $1`. Tenants cannot read, download, or delete files belonging to another organization. Before writing to disk, the gateway tracks both persisted organization byte totals and in-flight upload reservations so concurrent uploads cannot race past `server.max_file_storage_bytes_per_org`.
4. **Multipart Spill Cleanup**:
Temporary multipart spill files created during `ParseMultipartForm` are explicitly removed via `defer r.MultipartForm.RemoveAll()` on both success and error paths.
---
## File Schemas
### File Object Schema (`CanonicalFile`)
```json
{
"id": "file-89fa21bc-1240-4fe9-b001-c8120fa23e11",
"object": "file",
"bytes": 284102,
"created_at": 1740000000,
"filename": "batch_prompts_2026.jsonl",
"purpose": "batch",
"status": "uploaded",
"content_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
```
---
## Code Examples
#### cURL
```bash
# 1. Upload a JSONL file for Batch processing
curl -X POST http://localhost:8080/v1/files \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@requests.jsonl" \
-F "purpose=batch"
# 2. List all files for the organization
curl -X GET http://localhost:8080/v1/files \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
# 3. Retrieve file metadata
curl -X GET http://localhost:8080/v1/files/file-89fa21bc-1240-4fe9-b001-c8120fa23e11 \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
# 4. Stream / Download file content
curl -X GET http://localhost:8080/v1/files/file-89fa21bc-1240-4fe9-b001-c8120fa23e11/content \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-o downloaded_requests.jsonl
# 5. Delete file
curl -X DELETE http://localhost:8080/v1/files/file-89fa21bc-1240-4fe9-b001-c8120fa23e11 \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
# Upload File
file_record = client.files.create(
file=open("prompts.jsonl", "rb"),
purpose="batch"
)
print(f"Uploaded File ID: {file_record.id} ({file_record.bytes} bytes)")
# List Files
files = client.files.list()
for f in files.data:
print(f"- {f.id}: {f.filename} [{f.purpose}]")
# Retrieve Content
content = client.files.content(file_record.id)
with open("retrieved_prompts.jsonl", "wb") as f:
f.write(content.read())
# Delete File
client.files.delete(file_record.id)
print(f"File {file_record.id} deleted.")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"mime/multipart"
"net/http"
"os"
"time"
)
type FileRecord struct {
ID string `json:"id"`
Bytes int64 `json:"bytes"`
Filename string `json:"filename"`
Purpose string `json:"purpose"`
Status string `json:"status"`
SHA256 string `json:"content_sha256"`
}
func main() {
body := &bytes.Buffer{}
writer := multipart.NewWriter(body)
_ = writer.WriteField("purpose", "batch")
part, _ := writer.CreateFormFile("file", "eval_batch.jsonl")
_, _ = part.Write([]byte(`{"custom_id":"req-1","method":"POST","url":"/v1/chat/completions","body":{"model":"fast","messages":[{"role":"user","content":"Hello"}]}}` + "\n"))
_ = writer.Close()
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/files",
body,
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", writer.FormDataContentType())
client := &http.Client{Timeout: 30 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
respBytes, _ := io.ReadAll(resp.Body)
var fileRec FileRecord
_ = json.Unmarshal(respBytes, &fileRec)
fmt.Printf("File Uploaded: %s (Bytes: %d, SHA: %s)\n",
fileRec.ID, fileRec.Bytes, fileRec.SHA256)
}
```
---
## Asynchronous Batch Processing API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/batches
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/batches.md
Section: Control Plane Documentation > Use the API
# Asynchronous Batch Processing API
For high-volume, non-interactive workloads (such as dataset labeling, offline evaluation, bulk summarization, and synthetic data generation), GoRoute AI Gateway provides the **OpenAI-Compatible Batch Processing API**.
Batches decouple request ingestion from immediate execution. By accepting a 24-hour completion SLA, workloads run over off-peak provider headroom at up to a **50% discount** compared to synchronous on-demand inference.
---
## Endpoint Specification
| Endpoint | Method | Output | Description |
| :--- | :--- | :--- | :--- |
| `/v1/batches` | `POST` | `CanonicalBatch` JSON | Submit a new asynchronous batch job. |
| `/v1/batches` | `GET` | List JSON | List all recent batches for the organization. |
| `/v1/batches/{batch_id}` | `GET` | `CanonicalBatch` JSON | Inspect execution status and progress counts. |
| `/v1/batches/{batch_id}/cancel`| `POST` | `CanonicalBatch` JSON | Abort an ongoing or pending batch job. |
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Turnaround SLA (`completion_window`)**: `24h`
- **Supported Target Endpoint**: `/v1/chat/completions`
---
## State Machine Transitions
```mermaid
stateDiagram-v2
[*] --> validating: POST /v1/batches
validating --> in_progress: Valid input file & schema
validating --> failed: Invalid file or syntax error
in_progress --> completed: All records processed
in_progress --> cancelled: POST /v1/batches/{id}/cancel
in_progress --> failed: Disk or database failure
completed --> [*]
failed --> [*]
cancelled --> [*]
```
| State | Description |
| :--- | :--- |
| `validating` | Gateway checks tenant ownership of `input_file_id`, file purpose (`batch`), and verifies initial JSONL syntax. |
| `in_progress` | Background worker parses records sequentially, checks RBAC permissions per line item, and dispatches inference. |
| `completed` | All items have completed; output JSONL file is finalized and registered with `purpose: "batch_output"`. |
| `failed` | The batch encountered an unrecoverable failure (e.g. malformed lines exceeding buffer limits or storage loss). |
| `cancelled` | The caller explicitly aborted the batch before completion; completed items are preserved in output. |
---
## JSONL File Formats
### 1. Input File (`.jsonl`)
Upload using [`POST /v1/files`](https://goroute-ai.github.io/docs/gateway/endpoints/files) with `purpose="batch"`:
```json
{"custom_id": "eval-task-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "fast", "messages": [{"role": "user", "content": "Classify sentiment: Great latency."}]}}
{"custom_id": "eval-task-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "fast", "messages": [{"role": "user", "content": "Classify sentiment: Connection dropped."}]}}
```
### 2. Output File (`.jsonl`)
Once the batch reaches `status: "completed"`, download via `GET /v1/files/{output_file_id}/content`:
```json
{"id": "batch_req_01a", "custom_id": "eval-task-001", "response": {"status_code": 200, "request_id": "req_88f9...", "body": {"choices": [{"message": {"role": "assistant", "content": "Positive"}}]}}, "error": null}
{"id": "batch_req_02b", "custom_id": "eval-task-002", "response": {"status_code": 200, "request_id": "req_11a4...", "body": {"choices": [{"message": {"role": "assistant", "content": "Negative"}}]}}, "error": null}
```
---
## Code Examples
#### cURL
```bash
# 1. Upload input file
FILE_ID=$(curl -s -X POST http://localhost:8080/v1/files \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@batch_tasks.jsonl" \
-F "purpose=batch" | jq -r .id)
# 2. Submit Batch job
BATCH_ID=$(curl -s -X POST http://localhost:8080/v1/batches \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d "{
\"input_file_id\": \"$FILE_ID\",
\"endpoint\": \"/v1/chat/completions\",
\"completion_window\": \"24h\",
\"metadata\": {\"job_name\": \"nightly-evals\"}
}" | jq -r .id)
echo "Created batch: $BATCH_ID"
# 3. Check Batch Status
curl -X GET "http://localhost:8080/v1/batches/$BATCH_ID" \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
# 4. Cancel Batch (if needed)
curl -X POST "http://localhost:8080/v1/batches/$BATCH_ID/cancel" \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
#### Python (OpenAI SDK)
```python
import time
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
# 1. Upload file
file = client.files.create(
file=open("tasks.jsonl", "rb"),
purpose="batch"
)
# 2. Create batch
batch = client.batches.create(
input_file_id=file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"pipeline": "benchmark_v2"}
)
print(f"Batch {batch.id} submitted (status: {batch.status})")
# 3. Poll batch status
while True:
b = client.batches.retrieve(batch.id)
print(f"Status: {b.status} | Completed: {b.request_counts.completed}/{b.request_counts.total}")
if b.status in ("completed", "failed", "cancelled"):
if b.status == "completed":
print(f"Output File ID: {b.output_file_id}")
break
time.sleep(5)
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type CreateBatchRequest struct {
InputFileID string `json:"input_file_id"`
Endpoint string `json:"endpoint"`
CompletionWindow string `json:"completion_window"`
Metadata map[string]string `json:"metadata,omitempty"`
}
type BatchResponse struct {
ID string `json:"id"`
Status string `json:"status"`
OutputFileID string `json:"output_file_id,omitempty"`
RequestCounts struct {
Total int `json:"total"`
Completed int `json:"completed"`
Failed int `json:"failed"`
} `json:"request_counts"`
}
func main() {
payload, _ := json.Marshal(CreateBatchRequest{
InputFileID: "file-89fa21bc-1240-4fe9-b001-c8120fa23e11",
Endpoint: "/v1/chat/completions",
CompletionWindow: "24h",
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/batches",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 10 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var b BatchResponse
_ = json.Unmarshal(body, &b)
fmt.Printf("Batch created: %s | Status: %s\n", b.ID, b.Status)
}
```
---
## FinOps & Cost Economics
Because batch execution allows upstream providers to schedule processing during lower-demand periods, GoRoute applies the batch modality rate:
- **Up to 50% discount** relative to synchronous inference rates.
- Each completed request line item commits an atomic `SpendEvent` with `modality: "batch"` and exact token counts.
- Failed items do not incur completion token costs.
---
## Content Moderation API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/moderations
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/moderations.md
Section: Control Plane Documentation > Use the API
# Content Moderation API
GoRoute AI Gateway provides automated content moderation powered by OpenAI's `omni-moderation-latest` model. The Moderation endpoint analyzes input text for potentially harmful content across a broad taxonomy of safety categories, returning both boolean flags and granular confidence scores.
---
## Endpoint Specification
- **HTTP Method**: `POST`
- **Supported Paths**:
- `/v1/moderations`
- `/moderations` (Unversioned alias)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
- **Default Model**: `omni-moderation-latest`
---
## Safety & Fail-Closed Semantics
The moderation pipeline enforces strict enterprise safety guarantees:
1. **Fail-Closed Policy**: If moderation fails (network timeout, upstream error, or token budget exhaustion), the gateway does not silently permit the traffic to bypass inspection. Downstream applications can rely on explicit verdicts before presenting user input to generative models.
2. **RBAC Model Authorization**: The virtual key identity context is verified against org policy to ensure the caller has permissions to invoke moderation models.
3. **Multi-Input Batching**: Accepts either a single string or an array of strings in a single HTTP request for high-throughput ingress screening.
> [!IMPORTANT]
> Because safety classification occurs prior to expensive LLM reasoning passes, GoRoute allocates minimal rate limiter quota (`len(inputs) * 10` tokens) to ensure moderation calls are never throttled ahead of inference traffic.
---
## Request & Response Schemas
### Request Schema (`CanonicalModerationRequest`)
```json
{
"model": "omni-moderation-latest",
"input": [
"Here is an ordinary engineering question about distributed consensus.",
"Text to be evaluated for safety policy compliance."
]
}
```
| Field | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| `model` | string | No | The moderation model to use. Defaults to `omni-moderation-latest` if omitted. |
| `input` | string \| string[] | **Yes** | The input string or array of strings to classify for safety violations. |
### Response Schema (`CanonicalModerationResponse`)
```json
{
"id": "modr_34a9b2c8-89f1-419b-b9d2-9723ec0110fa",
"model": "omni-moderation-latest",
"provider": "openai",
"results": [
{
"flagged": false,
"categories": {
"harassment": false,
"harassment/threatening": false,
"hate": false,
"hate/threatening": false,
"illicit": false,
"illicit/violent": false,
"self-harm": false,
"self-harm/instructions": false,
"self-harm/intent": false,
"sexual": false,
"sexual/minors": false,
"violence": false,
"violence/graphic": false
},
"category_scores": {
"harassment": 0.00018,
"harassment/threatening": 0.00004,
"hate": 0.00008,
"hate/threatening": 0.00001,
"illicit": 0.00032,
"illicit/violent": 0.00002,
"self-harm": 0.00001,
"self-harm/instructions": 0.00001,
"self-harm/intent": 0.00001,
"sexual": 0.00012,
"sexual/minors": 0.00001,
"violence": 0.00045,
"violence/graphic": 0.00006
}
}
]
}
```
---
## Code Examples
#### cURL
```bash
# Single string input
curl -X POST http://localhost:8080/v1/moderations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"input": "Explain how to secure a Kubernetes cluster with mTLS."
}'
# Batch multi-string input with explicit model
curl -X POST http://localhost:8080/v1/moderations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "omni-moderation-latest",
"input": [
"Hello world, checking input moderation.",
"Another piece of candidate text."
]
}'
```
#### Python (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(
api_key="sk-gw-prod-7f9a2b8c4d1e",
base_url="http://localhost:8080/v1"
)
response = client.moderations.create(
model="omni-moderation-latest",
input=[
"Building resilient microservices using Go and Chi router.",
"A second candidate text for pre-flight validation."
]
)
for idx, result in enumerate(response.results):
print(f"Item {idx}: Flagged = {result.flagged}")
if result.flagged:
for category, is_violation in result.categories:
if is_violation:
score = getattr(result.category_scores, category, 0.0)
print(f" - Violated {category} (score: {score:.4f})")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type ModerationRequest struct {
Model string `json:"model,omitempty"`
Input []string `json:"input"`
}
type ModerationResponse struct {
ID string `json:"id"`
Model string `json:"model"`
Provider string `json:"provider"`
Results []struct {
Flagged bool `json:"flagged"`
Categories map[string]bool `json:"categories"`
CategoryScores map[string]float64 `json:"category_scores"`
} `json:"results"`
}
func main() {
payload, _ := json.Marshal(ModerationRequest{
Model: "omni-moderation-latest",
Input: []string{
"Architectural guidelines for GoRoute AI Gateway.",
},
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/moderations",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var modResp ModerationResponse
_ = json.Unmarshal(body, &modResp)
fmt.Printf("Moderation ID: %s | Provider: %s\n", modResp.ID, modResp.Provider)
for i, r := range modResp.Results {
fmt.Printf(" Item %d: Flagged = %t\n", i, r.Flagged)
}
}
```
---
## Response Headers
| Header | Description |
| :--- | :--- |
| `X-Gateway-Request-Id` | Unique UUID tracking the moderation lifecycle (`req_...`) |
| `X-Gateway-Provider` | Upstream provider executing classification (`openai`) |
| `X-Gateway-Cost-Micros` | FinOps charge incurred (omni-moderation is zero-cost upstream, billed at 0 micro-dollars) |
| `X-Gateway-Pricing-Status` | Provenance status (`verified`) |
---
## Fast Heuristic Tokenization API
URL: https://goroute-ai.github.io/docs/gateway/endpoints/tokenize
Markdown: https://goroute-ai.github.io/docs/gateway/endpoints/tokenize.md
Section: Control Plane Documentation > Use the API
# Fast Heuristic Tokenization API
Accurate token estimation is vital for client-side budgeting, chunking text for RAG (Retrieval-Augmented Generation), and pre-flight validation before executing long-context inference.
GoRoute AI Gateway includes a **zero-latency, in-process subword tokenizer** accessible via `POST /v1/tokenize`. Unlike traditional proxies that make network roundtrips to an external tokenizer service or invoke heavyweight Cgo Python bindings (like `tiktoken`), GoRoute runs an optimized, pure-Go heuristic scanner that approximates Byte Pair Encoding (BPE) and SentencePiece vocabularies in under **50 microseconds**.
---
## Endpoint Specification
- **HTTP Method**: `POST`
- **Supported Paths**:
- `/v1/tokenize`
- `/tokenize` (Unversioned alias)
- **Authentication**: `Bearer sk-gw-...` (Virtual Key or Master Admin Key)
- **Content-Type**: `application/json`
---
## Tokenizer Profiles & Vocabulary Heuristics
The gateway dynamically resolves models to one of five specialized tokenization profiles (`internal/ratelimit/tokenizer.go`):
| Profile | Applicable Models | Vocabulary Divisor | Description |
| :--- | :--- | :--- | :--- |
| `o200k_base` | GPT-4o, GPT-4.1, o1, o3, o4 | `4.2` chars/token | 200k vocabulary; high compression on multilingual & code. |
| `cl100k_base` | GPT-4, GPT-3.5, `text-embedding-3-*` | `3.9` chars/token | Standard 100k OpenAI BPE vocabulary. |
| `claude_bpe` | Claude 3.5 Sonnet, Claude 3.7, Claude 3 Opus | `3.6` chars/token | Denser subword encoding on prose and system instructions. |
| `gemini_sp` | Gemini 2.5 Flash, Gemini 2.5 Pro, Gemma | `4.1` chars/token | Google SentencePiece unigram/BPE mixture. |
| `mistral_tekken`| Mistral Large, Codestral, Llama 3/3.3 | `3.8` chars/token | Tekken tokenizer with expanded code character tables. |
### Mathematical Estimation Algorithm
For any given input text, the scanner decomposes the byte stream across three dimensions:
```text
Tokens = ⌈(AlnumChars * 100) / Divisor⌉ + ⌊(PunctChars + 1) / 2⌋ + ⌊(UTF8NonASCIIBytes + 1) / 2⌋
```
1. **Alphanumeric Characters**: Scaled using the family-specific character-to-token ratio.
2. **Punctuation & Code Syntax**: Code brackets (`{`, `}`, `[`, `]`), JSON quotes, and operators often form dedicated BPE tokens.
3. **Multi-Byte UTF-8 Bytes**: Multi-byte sequences (emojis, Cyrillic, CJK, Arabic) undergo multi-step BPE merges.
---
## Request & Response Schemas
### Request Schema
Supports either a raw text prompt, or structured chat messages and tool definitions:
```json
{
"model": "gpt-4.1",
"prompt": "Explain Byzantine fault tolerance in distributed databases."
}
```
Or for structured multi-turn conversations:
```json
{
"model": "claude-3-7-sonnet",
"messages": [
{"role": "system", "content": "You are a concise code reviewer."},
{"role": "user", "content": "Review this pull request diff."}
],
"tools": [
{
"type": "function",
"function": {
"name": "lookup_git_diff",
"description": "Fetch PR diff by commit hash",
"parameters": {
"type": "object",
"properties": {
"commit": {"type": "string"}
}
}
}
}
]
}
```
### Response Schema
```json
{
"model": "gpt-4.1",
"profile": "o200k_base",
"count": 9,
"total_tokens": 9
}
```
---
## Code Examples
#### cURL
```bash
# Tokenize a raw prompt
curl -X POST http://localhost:8080/v1/tokenize \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4.1",
"prompt": "GoRoute AI Gateway performs zero-allocation subword tokenization."
}'
# Tokenize full chat messages using Claude BPE profile
curl -X POST http://localhost:8080/v1/tokenize \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-3-5-sonnet",
"messages": [
{"role": "system", "content": "You are an AI assistant."},
{"role": "user", "content": "How do goroutines manage call frames?"}
]
}'
```
#### Python
```python
import httpx
client = httpx.Client(
base_url="http://localhost:8080",
headers={"Authorization": "Bearer sk-gw-prod-7f9a2b8c4d1e"}
)
response = client.post("/v1/tokenize", json={
"model": "gemini-2.5-flash",
"prompt": "Estimate token requirements for RAG context window budgeting."
})
result = response.json()
print(f"Profile: {result['profile']}")
print(f"Token count: {result['count']}")
```
#### Go
```go
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type TokenizeRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
}
type TokenizeResponse struct {
Model string `json:"model"`
Profile string `json:"profile"`
Count int `json:"count"`
TotalTokens int `json:"total_tokens"`
}
func main() {
payload, _ := json.Marshal(TokenizeRequest{
Model: "claude-3-7-sonnet",
Prompt: "Micro-dollar spend calculations with ceiling integer division.",
})
req, _ := http.NewRequestWithContext(
context.Background(),
http.MethodPost,
"http://localhost:8080/v1/tokenize",
bytes.NewReader(payload),
)
req.Header.Set("Authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
req.Header.Set("Content-Type", "application/json")
client := &http.Client{Timeout: 3 * time.Second}
resp, err := client.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
var tokResp TokenizeResponse
_ = json.Unmarshal(body, &tokResp)
fmt.Printf("Model: %s | Profile: %s | Tokens: %d\n",
tokResp.Model, tokResp.Profile, tokResp.Count)
}
```
---
## Comparison: In-Process Heuristic vs Network Tokenizer
| Feature | GoRoute `/v1/tokenize` | Remote Tokenizer Service | Cgo / Tiktoken Binding |
| :--- | :--- | :--- | :--- |
| **Execution Latency** | **< 0.05 ms** (in-process) | 15 – 50 ms (HTTP network) | 0.5 – 2 ms |
| **Memory Allocation** | Zero heap allocation | TCP socket buffers | Cgo stack transition overhead |
| **Network Failure Mode** | None (always available) | External point of failure | Process segmentation risk |
| **Cross-Provider Profiles**| Auto-adapts to GPT, Claude, Gemini, Mistral | Usually single vendor only | Requires multiple native libraries |
> [!NOTE]
> The in-process token counter is also invoked automatically on every incoming `/v1/chat/completions` request for pre-flight rate limiting and sliding-window quota reservations.
---
## Providers Overview
URL: https://goroute-ai.github.io/docs/providers/overview
Markdown: https://goroute-ai.github.io/docs/providers/overview.md
Section: Control Plane Documentation > Choose a model provider
# Providers Overview
GoRoute connects applications to different hosted and local AI providers through a consistent gateway interface. This means your application can keep using the same connection while you change or add providers behind it. The catalog lists supported provider integrations; model names and availability depend on the provider and your GoRoute configuration.
The catalog includes **73 `ProviderType` profiles** plus `CustomOpenAICompatible` endpoints. GoRoute handles the protocol translation, authentication, credential storage, retries, and usage tracking needed for each supported integration.
```mermaid
flowchart TD
Client["Client Application / SDK"] -->|Standard OpenAI Protocol| GW["GoRoute AI Gateway"]
subgraph CoreEngine["GoRoute Core Engine"]
Router["Model Router & Failover"]
Ledger["FinOps Cost & Token Ledger"]
SSRF["SSRF Network Hardening"]
end
GW --> CoreEngine
CoreEngine --> A1["Native OpenAI Adapter\n(Chat, reasoning, audio, images, embeddings)"]
CoreEngine --> A2["Native Anthropic Adapter\n(Claude Messages API)"]
CoreEngine --> A3["Native Gemini & Vertex Adapters\n(Gemini API, multimodal, Imagen)"]
CoreEngine --> A4["Native AWS Bedrock Adapter\n(SigV4, foundation model APIs)"]
CoreEngine --> A5["Native Azure OpenAI Adapter\n(API Key, Entra ID, Version Pinning)"]
CoreEngine --> A6["Native Cohere Adapter\n(Command R+, Embed v3, Rerank v3.5)"]
CoreEngine --> A7["OpenAI-Compatible Dialect Engine\n(60+ Built-in & Extended Profiles: Cerebras, Groq, DeepSeek, Together, vLLM, etc.)"]
```
---
## Interactive Provider Catalog
Filter all **73 `ProviderType` profiles** and `CustomOpenAICompatible` by adapter tier, category, or capability, and click any provider ID to copy it for configuration.
### Supported Provider Profiles (74 Built-in)
| ID | Provider | Tier | Category | Auth | Capabilities | Default Base URL |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| `openai` | OpenAI | Native Adapter | Cloud Flagship | Bearer Token | chat, streaming, tools, vision, reasoning, embeddings, structuredOutputs, audio, images | `https://api.openai.com/v1` |
| `anthropic` | Anthropic Claude | Native Adapter | Cloud Flagship | x-api-key | chat, streaming, tools, vision, reasoning, structuredOutputs | `https://api.anthropic.com/v1` |
| `gemini` | Google Gemini | Native Adapter | Cloud Flagship | API Key / OAuth | chat, streaming, tools, vision, reasoning, embeddings, structuredOutputs | `https://generativelanguage.googleapis.com/v1beta` |
| `mistral` | Mistral AI | Native Adapter | Cloud Flagship | Bearer Token | chat, streaming, tools, vision, embeddings, structuredOutputs | `https://api.mistral.ai/v1` |
| `ollama` | Ollama (Local) | Native Adapter | Local & Self-Hosted | None (Local) | chat, streaming, tools, vision, embeddings, structuredOutputs | `http://localhost:11434` |
| `bedrock` | AWS Bedrock | Native Adapter | Cloud Flagship | AWS SigV4 | chat, streaming, tools, vision, reasoning, embeddings | `https://bedrock-runtime..amazonaws.com` |
| `vertex` | Google Vertex AI | Native Adapter | Cloud Flagship | GCP Service Account / OAuth2 | chat, streaming, tools, vision, reasoning, embeddings, structuredOutputs | `https://-aiplatform.googleapis.com` |
| `cohere` | Cohere | Native Adapter | Cloud Flagship | Bearer Token | chat, streaming, tools, embeddings, rerank, structuredOutputs | `https://api.cohere.com/v2` |
| `groq` | Groq Fast Inference | Launch Set | Fast Inference | Bearer Token | chat, streaming, tools, vision, structuredOutputs, audio | `https://api.groq.com/openai/v1` |
| `deepseek` | DeepSeek AI | Launch Set | Cloud Flagship | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.deepseek.com/v1` |
| `together` | Together AI | Launch Set | Fast Inference | Bearer Token | chat, streaming, tools, embeddings, rerank, structuredOutputs | `https://api.together.xyz/v1` |
| `azure` | Microsoft Azure OpenAI | Launch Set | Cloud Flagship | api-key / Entra ID | chat, streaming, tools, vision, embeddings, structuredOutputs | `https://.openai.azure.com` |
| `cerebras` | Cerebras Wafer-Scale Engine | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.cerebras.ai/v1` |
| `sambanova` | SambaNova Systems | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.sambanova.ai/v1` |
| `xai` | xAI (Grok) | Built-in Dialect | Cloud Flagship | Bearer Token | chat, streaming, tools, vision, reasoning, structuredOutputs, images, audio | `https://api.x.ai/v1` |
| `fireworks` | Fireworks AI | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, vision, embeddings, rerank, structuredOutputs, images, audio | `https://api.fireworks.ai/inference/v1` |
| `perplexity` | Perplexity AI | Built-in Dialect | Specialized & Embeddings | Bearer Token | chat, streaming | `https://api.perplexity.ai` |
| `anyscale` | Anyscale Endpoints | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, embeddings, structuredOutputs | `https://api.endpoints.anyscale.com/v1` |
| `deepinfra` | DeepInfra | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, embeddings, structuredOutputs | `https://api.deepinfra.com/v1/openai` |
| `novita` | Novita AI | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, embeddings, rerank, structuredOutputs | `https://api.novita.ai/v3/openai` |
| `nebius` | Nebius AI Studio | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.studio.nebius.ai/v1` |
| `hyperbolic` | Hyperbolic AI | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.hyperbolic.xyz/v1` |
| `lambda` | Lambda Labs Inference | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools | `https://api.lambdalabs.com/v1` |
| `friendli` | Friendli AI | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://inference.friendli.ai/v1` |
| `openrouter` | OpenRouter | Built-in Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, vision, reasoning, structuredOutputs | `https://openrouter.ai/api/v1` |
| `vllm` | vLLM Local / Self-Hosted | Built-in Dialect | Local & Self-Hosted | Bearer Token (Optional) | chat, streaming, tools, embeddings, structuredOutputs | `http://localhost:8000/v1` |
| `cloudflare` | Cloudflare Workers AI | Built-in Dialect | Cloud Flagship | Bearer Token | chat, streaming, tools, embeddings, audio | `https://api.cloudflare.com/client/v4/ai/v1` |
| `databricks` | Databricks Model Serving | Built-in Dialect | Cloud Flagship | Bearer Token | chat, streaming, tools, embeddings, structuredOutputs | `https://.cloud.databricks.com/serving-endpoints` |
| `snowflake` | Snowflake Cortex REST API | Built-in Dialect | Cloud Flagship | Bearer Token / Key-Pair JWT | chat, streaming, tools, embeddings | `https://.snowflakecomputing.com/api/v2/cortex/inference` |
| `dashscope` | Alibaba Cloud DashScope (Qwen) | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools, vision, embeddings, structuredOutputs | `https://dashscope.aliyuncs.com/compatible-mode/v1` |
| `volcengine` | ByteDance Volcengine (Doubao) | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools, vision, reasoning, structuredOutputs | `https://ark.cn-beijing.volces.com/api/v3` |
| `bigmodel` | Zhipu AI BigModel (GLM) | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools, vision, embeddings, structuredOutputs | `https://open.bigmodel.cn/api/paas/v4` |
| `moonshot` | Moonshot AI (Kimi) | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.moonshot.cn/v1` |
| `minimax` | MiniMax | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.minimax.chat/v1` |
| `siliconflow` | SiliconFlow | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools, reasoning, embeddings, structuredOutputs | `https://api.siliconflow.cn/v1` |
| `voyage` | Voyage AI | Built-in Dialect | Specialized & Embeddings | Bearer Token | embeddings, rerank | `https://api.voyageai.com/v1` |
| `jina` | Jina AI | Built-in Dialect | Specialized & Embeddings | Bearer Token | embeddings, rerank | `https://api.jina.ai/v1` |
| `replicate` | Replicate Deployments | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools | `https://api.replicate.com/v1` |
| `ai21` | AI21 Labs Studio | Built-in Dialect | Cloud Flagship | Bearer Token | chat, streaming, tools | `https://api.ai21.com/studio/v1` |
| `baseten` | Baseten Model Gateway | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools | `https://bridge.baseten.co/v1` |
| `nvidia` | NVIDIA NIM APIs | Built-in Dialect | Fast Inference | Bearer Token | chat, streaming, tools, vision, rerank, structuredOutputs | `https://integrate.api.nvidia.com/v1` |
| `scaleway` | Scaleway Generative APIs | Built-in Dialect | Regional & International | Bearer Token | chat, streaming, tools | `https://api.scaleway.ai/v1` |
| `github_copilot` | GitHub Copilot | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.githubcopilot.com` |
| `vercel_ai_gateway` | Vercel AI Gateway | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, reasoning, embeddings, structuredOutputs | `https://ai-gateway.vercel.sh/v1` |
| `huggingface` | Hugging Face Inference | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, embeddings, structuredOutputs | `https://router.huggingface.co/v1` |
| `featherless` | Featherless AI | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.featherless.ai/v1` |
| `fal` | fal.ai | Extended Dialect | Specialized & Embeddings | Bearer Token | chat, streaming, tools, structuredOutputs, images | `https://fal.run/openrouter/router/openai/v1` |
| `zai` | Z.ai | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.z.ai/api/paas/v4` |
| `poe` | Poe | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.poe.com/v1` |
| `helicone` | Helicone AI Gateway | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, structuredOutputs | `https://ai-gateway.helicone.ai/v1` |
| `aihubmix` | AIHubMix | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, reasoning, embeddings, structuredOutputs | `https://aihubmix.com/v1` |
| `venice` | Venice AI | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.venice.ai/api/v1` |
| `chutes` | Chutes AI | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://llm.chutes.ai/v1` |
| `gmi` | GMI Cloud | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.gmi-serving.com/v1` |
| `llamagate` | LlamaGate | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.llamagate.dev/v1` |
| `sarvam` | Sarvam AI | Extended Dialect | Regional & International | api-subscription-key | chat, streaming, tools, structuredOutputs | `https://api.sarvam.ai/v1` |
| `crusoe` | Crusoe Cloud | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://managed-inference.crusoecloud.com/v1` |
| `parasail` | Parasail | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.parasail.io/v1` |
| `upstage` | Upstage Solar | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, embeddings, structuredOutputs | `https://api.upstage.ai/v1/solar` |
| `ovhcloud` | OVHcloud AI Endpoints | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, embeddings, structuredOutputs | `https://oai.endpoints.kepler.ai.cloud.ovh.net/v1` |
| `wandb` | W&B Inference | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.inference.wandb.ai/v1` |
| `nscale` | Nscale | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, structuredOutputs | `https://inference.api.nscale.com/v1` |
| `publicai` | Public AI | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.publicai.co/v1` |
| `synthetic` | Synthetic | Extended Dialect | Fast Inference | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.synthetic.new/openai/v1` |
| `nano_gpt` | NanoGPT | Extended Dialect | Aggregators & Gateways | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://nano-gpt.com/api/v1` |
| `xiaomi_mimo` | Xiaomi MiMo | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, reasoning, structuredOutputs | `https://api.xiaomimimo.com/v1` |
| `reka` | Reka AI | Extended Dialect | Cloud Flagship | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.reka.ai/v1` |
| `yi` | 01.AI (Yi) | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.lingyiwanwu.com/v1` |
| `baichuan` | Baichuan AI | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.baichuan-ai.com/v1` |
| `stepfun` | StepFun | Extended Dialect | Regional & International | Bearer Token | chat, streaming, tools, structuredOutputs | `https://api.stepfun.com/v1` |
| `lm_studio` | LM Studio (Local) | Extended Dialect | Local & Self-Hosted | Bearer Token (Optional) | chat, streaming, tools, reasoning, embeddings, structuredOutputs | `http://localhost:1234/v1` |
| `llamafile` | llamafile (Local) | Extended Dialect | Local & Self-Hosted | Bearer Token (Optional) | chat, streaming, tools, embeddings, structuredOutputs | `http://localhost:8080/v1` |
| `xinference` | Xorbits Inference | Extended Dialect | Local & Self-Hosted | Bearer Token (Optional) | chat, streaming, tools, embeddings, structuredOutputs | `http://localhost:9997/v1` |
| `custom_openai_compatible` | Custom OpenAI-Compatible | Custom | Local & Self-Hosted | Configurable (Bearer / Custom Header) | chat, streaming, tools, embeddings, structuredOutputs | `https:///v1` |
---
## Provider Integration Matrix (73 ProviderType Profiles)
GoRoute registers **73 `ProviderType` profiles** in `AllProviders()` (`internal/domain/inference.go`), combining dedicated native protocol adapters (`openai`, `anthropic`, `gemini`, `bedrock`, `vertex`, `azure`, `cohere`, `mistral`, `ollama`), **60+ certified OpenAI-compatible dialect profiles** (`internal/adapter/openaicompat/builtin_profiles.go` and `extended_profiles.go`), and `CustomOpenAICompatible` for arbitrary self-hosted or private endpoints.
This matrix highlights core native adapters and representative dialect integrations. Model IDs change more often than adapters do, so model names shown in this guide and linked provider pages are representative examples, not an exhaustive or guaranteed-current inventory.
| Provider | Adapter Type | Supported Modalities | Streaming | Tools | Auth Strategy | Guide |
| :--- | :--- | :--- | :---: | :---: | :--- | :--- |
| **OpenAI** | Native Adapter | Chat, Embed, Moderation, Audio, Images, Batches, Realtime | Yes | Yes | Bearer Token (`OPENAI_API_KEY`) | [OpenAI](https://goroute-ai.github.io/docs/providers/openai) |
| **Anthropic** | Native Messages | Chat | Yes | Yes | `x-api-key` (`ANTHROPIC_API_KEY`) | [Anthropic Claude](https://goroute-ai.github.io/docs/providers/anthropic-claude) |
| **Google Gemini** | Native Gemini | Chat, Embed, Multimodal Audio/Vision, Imagen | Yes | Yes | `x-goog-api-key` (`GEMINI_API_KEY`) | [Google Gemini](https://goroute-ai.github.io/docs/providers/google-gemini) |
| **AWS Bedrock** | Native Converse | Chat, Embed, Tools | Yes (EventStream) | Yes | AWS SigV4 HMAC-SHA256 | [AWS Bedrock](https://goroute-ai.github.io/docs/providers/aws-bedrock) |
| **Google Vertex AI** | Native Vertex | Chat, Embed, Multimodal | Yes (SSE) | Yes | GCP SA JSON / OAuth2 Bearer | [Google Vertex](https://goroute-ai.github.io/docs/providers/google-vertex) |
| **Azure OpenAI** | Native Azure | Chat, Embed, Vision | Yes | Yes | `api-key` / Entra ID Bearer | [Azure OpenAI](https://goroute-ai.github.io/docs/providers/azure-openai) |
| **Cohere** | Native Cohere v2 | Chat, Embed, Rerank | Yes | Yes | Bearer Token (`COHERE_API_KEY`) | [Cohere](https://goroute-ai.github.io/docs/providers/cohere) |
| **Mistral AI** | Native Adapter | Chat, Embed, Tools | Yes | Yes | Bearer Token (`MISTRAL_API_KEY`) | [Mistral AI](https://goroute-ai.github.io/docs/providers/mistral) |
| **Ollama** | Native Local | Chat, Embed | Yes (NDJSON) | No | None / Base URL (`http://localhost:11434`) | [Ollama](https://goroute-ai.github.io/docs/providers/ollama) |
| **Groq** | Certified Dialect | Chat, Vision | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#1-ultra-fast-wafer-scale--hardware-acceleration) |
| **DeepSeek** | Certified Dialect | Chat, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#3-search--reasoning) |
| **Together AI** | Certified Dialect | Chat, Embed, Vision | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Cerebras** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#1-ultra-fast-wafer-scale--hardware-acceleration) |
| **SambaNova** | Certified Dialect | Chat, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#1-ultra-fast-wafer-scale--hardware-acceleration) |
| **xAI (Grok)** | Certified Dialect | Chat, Vision, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#3-search--reasoning) |
| **Fireworks AI** | Certified Dialect | Chat, Embed, Vision | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Perplexity AI**| Certified Dialect | Chat, Search Citations | Yes | No | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#3-search--reasoning) |
| **Anyscale** | Certified Dialect | Chat, Embed | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#4-multi-model-aggregators) |
| **DeepInfra** | Certified Dialect | Chat, Embed | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Novita AI** | Certified Dialect | Chat, Embed, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Nebius AI Studio**| Certified Dialect | Chat, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Hyperbolic** | Certified Dialect | Chat, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Lambda Labs** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **Friendli AI** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
| **OpenRouter** | Certified Dialect | Chat, Vision, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#4-multi-model-aggregators) |
| **vLLM** | Certified Dialect | Chat, Embed | Yes | Yes | None / Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#5-self-hosted--local-inference-engines) |
| **Cloudflare** | Certified Dialect | Chat, Embed | Yes | Yes | Cloudflare API Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **Databricks** | Certified Dialect | Chat, Embed | Yes | Yes | Databricks Personal Access Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **Snowflake Cortex** | Certified Dialect | Chat, Embed | Yes | Yes | Snowflake JWT / Session Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **DashScope (Qwen)** | Certified Dialect | Chat, Embed, Vision | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **Volcengine** | Certified Dialect | Chat, Vision, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **BigModel (GLM)**| Certified Dialect | Chat, Embed, Vision | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **Moonshot (Kimi)** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **MiniMax** | Certified Dialect | Chat, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#8-global-enterprise--regional-providers) |
| **SiliconFlow** | Certified Dialect | Chat, Embed, Reasoning | Yes (Reasoning) | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#3-search--reasoning) |
| **Voyage AI** | Certified Dialect | Embeddings | No | No | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#6-specialized-embeddings--reranking) |
| **Jina AI** | Certified Dialect | Embeddings, Rerank | No | No | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#6-specialized-embeddings--reranking) |
| **Replicate** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#7-enterprise-foundation-model-platforms) |
| **AI21 Labs** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#7-enterprise-foundation-model-platforms) |
| **Baseten** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#5-self-hosted--local-inference-engines) |
| **NVIDIA NIM** | Certified Dialect | Chat, Vision | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#7-enterprise-foundation-model-platforms) |
| **Scaleway** | Certified Dialect | Chat | Yes | Yes | Bearer Token | [Dialects](https://goroute-ai.github.io/docs/providers/openai-compatible#2-high-performance-serverless-inference) |
### Find model IDs available to your key
Use the provider's current model documentation to choose an ID, then check the gateway's tenant-filtered `GET /v1/models` inventory with the same virtual key your application will use. That inventory is the reference for models currently exposed to that caller. Provider access, gateway catalog discovery/sync settings, organization policies, and key allowlists can all affect availability. A model listed by a provider is not by itself confirmation that a particular GoRoute deployment can route it.
When catalog discovery is enabled, the gateway can synchronize models exposed by supported providers. Discovery does not make every provider model universally available: provider credentials, adapter/protocol compatibility, organization policy, and virtual-key permissions still apply. See [Model Management & Access Control](https://goroute-ai.github.io/docs/gateway/endpoints/model-management) and [model catalog configuration](https://goroute-ai.github.io/docs/deploy/configuration).
### Embedded Model Catalog & Capability Generalizations (v1.6.0)
GoRoute embeds a curated catalog of **4,627 models** (`internal/modelcatalog/catalog.json.gz`) with operation-level support and modality pricing provenance. When a model is not yet in the catalog or has `CapabilityUnknown` fields, `internal/modelcatalog/generalizations.go` (`InferFamilyCapabilities`, `BackfillCapabilities`, `StripRegionalModelPrefix`) supplies explicit heuristic defaults (`Inferred: true`, `Source: "family-pattern-heuristic"`) for context windows and capabilities (tools, streaming, vision, reasoning, structured outputs) without treating unverified catalog rates as verified for budgeted traffic.
### Operation-Level Routing Enforcement (`SupportedOperations`)
In v1.6.0, `checkDeploymentOperation` (`internal/router/reliability.go`) enforces operation compatibility before dispatching a request:
- **Adapter-level check**: Adapters implementing `OperationSupportChecker.SupportsOperation(op)` reject unsupported operations with HTTP `501 Not Implemented` (`_not_supported`).
- **Model-level check**: When `GlobalModel.SupportedOperations` is populated, requests for operations outside that allowlist are rejected with HTTP `400 Bad Request` (`unsupported_operation`).
---
## Authentication Strategies
GoRoute abstracts vendor authentication into four distinct automated mechanisms:
1. **Standard Bearer Authentication (`BearerAuthStrategy`)**: Injects `Authorization: Bearer ` automatically for 35+ providers.
2. **Cryptographic AWS SigV4 (`SigV4Signer`)**: Signs requests dynamically using IAM access keys, temporary session tokens, and regional HMAC-SHA256 calculations.
3. **Google OAuth2 Assertion Exchange (`TokenSource`)**: Signs RS256 JWT assertions with GCP Service Account private keys, exchanges them for access tokens with `oauth2.googleapis.com`, and handles background caching.
4. **Microsoft Entra ID & API Key (`AzureAuthStrategy`)**: Automatically distinguishes between standard Azure resource keys (`api-key: `) and Entra ID tokens (`Authorization: Bearer `).
---
## Dynamic Routing & Model Aliases
Configure providers in `configs/config.yaml` using either explicit provider prefixes or virtual alias routing:
### 1. Direct Dynamic Prefix Routing
Call upstream providers directly by prefixing the model parameter:
```bash
# Routes directly to Cerebras Wafer-Scale Engine
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{"model": "cerebras/llama3.3-70b", "messages": [{"role": "user", "content": "Hello"}]}'
# Routes directly to Anthropic Claude 3.5 Sonnet
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{"model": "anthropic/claude-sonnet-5-5", "messages": [{"role": "user", "content": "Hello"}]}'
```
### 2. High-Availability Model Groups & Fallback Topologies
Define resilient model aliases in `configs/config.yaml`:
```yaml
model_list:
# Fast Tier: Gemini 2.5 Flash with fallback to GPT-4o-mini
- model_name: fast
routing_strategy: weighted-shuffle
fallbacks: ["fast-backup"]
model_params:
model: gemini/gemini-2.5-flash
api_key: os.environ/GEMINI_API_KEY
weight: 100
- model_name: fast-backup
model_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
# Reasoning Tier: Anthropic Claude Sonnet with fallback to Azure GPT-4o
- model_name: reasoning
routing_strategy: least-busy
fallbacks: ["azure-gpt4o"]
model_params:
model: anthropic/claude-sonnet-5-5
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: azure-gpt4o
model_params:
model: azure/gpt-4o
api_key: os.environ/AZURE_OPENAI_API_KEY
api_base: "https://my-enterprise.openai.azure.com"
azure_deployment_name: "gpt-4o-prod"
api_version: "2024-08-01-preview"
```
---
## Standard Response Telemetry
Every request processed across any of the 73 `ProviderType` profiles includes standardized audit and FinOps telemetry headers:
| Header | Description | Example |
| :--- | :--- | :--- |
| `X-Gateway-Request-Id` | Unique UUID assigned to the client invocation | `req_01h8a...` |
| `X-Trace-Id` | Distributed W3C trace ID for end-to-end tracing | `4bf92f3577b34da6a3ce929d0e0e4736` |
| `X-Gateway-Provider` | The actual upstream provider that fulfilled the request | `bedrock`, `cerebras`, `openai` |
| `X-Gateway-Model` | The upstream foundation model executed | `anthropic.claude-sonnet-5-5` |
| `X-Gateway-Cost-Micros`| Exact cost of the invocation in micro-dollars ($1 = 1,000,000 µ$) | `1250` ($0.001250) |
| `X-Gateway-Routing-Time`| Gateway latency overhead before upstream dispatch | `4.2ms` |
| `X-Gateway-Cache` | Cache hit status (`HIT` or `MISS`) | `HIT` |
| `Retry-After` | Cooldown period emitted during upstream 429 rate limit events | `60` |
---
## OpenAI
URL: https://goroute-ai.github.io/docs/providers/openai
Markdown: https://goroute-ai.github.io/docs/providers/openai.md
Section: Control Plane Documentation > Choose a model provider
# OpenAI
GoRoute provides a native adapter for **OpenAI**, with gateway routes for chat, embeddings, audio, images, moderation, Responses, and Realtime. Model IDs used in examples are provider-side examples, not an exhaustive or guaranteed-current list of models supported by every GoRoute deployment. OpenAI changes model IDs and endpoint availability over time; check the [OpenAI model catalog](https://developers.openai.com/api/docs/models) and then confirm the IDs exposed to your key with [`GET /v1/models`](https://goroute-ai.github.io/docs/api/gateway-api#4-model-catalogue-inventory-get-v1models).
---
## Modalities & Route Map
| Modality | Gateway Route | Upstream Engine |
| :--- | :--- | :--- |
| **Chat Completions** | `POST /v1/chat/completions` | Standard Chat REST |
| **Reasoning Models** | `POST /v1/chat/completions` | Reasoning Normalizer |
| **Embeddings** | `POST /v1/embeddings` | Vector Engine |
| **Audio Transcription** | `POST /v1/audio/transcriptions` | Audio transcription |
| **Audio Speech** | `POST /v1/audio/speech` | Speech generation |
| **Image Generation** | `POST /v1/images/generations` | Image generation |
| **Content Moderation**| `POST /v1/moderations` | Moderation classifier |
| **Responses API** | `POST /v1/responses` | Structured Responses |
| **Realtime WebSockets**| `GET /v1/realtime` | Full-duplex WebSockets |
OpenAI's current catalog includes IDs such as `gpt-6-astra` and `gpt-6.1-sol`; these are provider-side examples only. The model catalog describes endpoint availability per model, so do not assume that every listed model works with Chat Completions or every GoRoute route. Verify provider endpoint compatibility and your tenant-filtered gateway inventory before configuring a deployment.
---
## Reasoning Effort Normalization (o1 & o3)
OpenAI's reasoning models (`o1`, `o1-mini`, `o3`, `o3-mini`) enforce different API constraints than standard models. GoRoute's native adapter automatically handles these nuances:
1. **System Prompt Normalization**: OpenAI reasoning models reject messages with `role: "system"`. GoRoute transparently converts them to `role: "developer"` when targeting any model prefixed with `o1` or `o3`.
2. **Temperature and Top-P Stripping**: Because reasoning models enforce internal temperature controls, supplying `temperature` or `top_p` triggers an upstream `400 Bad Request`. GoRoute automatically strips these fields before transmission.
3. **Reasoning Effort Mapping**: Pass `reasoning_effort` (`"low"`, `"medium"`, `"high"`) directly in your request payload to control internal chain-of-thought token allocation:
```json
{
"model": "o3-mini",
"reasoning_effort": "high",
"messages": [
{"role": "developer", "content": "You are an expert algorithmic systems engineer."},
{"role": "user", "content": "Prove that 2-SAT can be solved in linear time using strongly connected components."}
]
}
```
---
## Audio Services: Whisper & TTS
### Audio Transcription (Whisper STT)
Available via `POST /v1/audio/transcriptions` and `/translations`. To protect gateway memory under concurrent uploads of large audio files (up to 25 MB), GoRoute uses **multipart disk spooling** (`internal/server/handlers_audio.go`):
- Audio payloads are spooled to temporary files with secure `0600` permissions.
- Zero-memory retryability: upstream network retries re-stream from disk rather than buffering gigabytes in RAM.
- Billing is tracked strictly per integer duration second.
```bash
curl -X POST http://localhost:8080/v1/audio/transcriptions \
-H "Authorization: Bearer your-virtual-key" \
-F "file=@meeting_recording.mp3" \
-F "model=whisper-1" \
-F "language=en"
```
### Text-to-Speech (TTS)
Available via `POST /v1/audio/speech`. GoRoute streams raw audio bytes back to the caller as they arrive from OpenAI:
- Supported formats: `mp3`, `opus`, `aac`, `flac`, `wav`, `pcm`.
- Voices: `alloy`, `echo`, `fable`, `onyx`, `nova`, `shimmer`.
- Billed per 1,000 characters using 64-bit ceiling integer division in the FinOps ledger.
```bash
curl -X POST http://localhost:8080/v1/audio/speech \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "tts-1",
"input": "GoRoute provides zero-overhead intelligent routing for modern enterprise AI.",
"voice": "nova"
}' --output speech.mp3
```
---
## Image Generation (DALL-E 2 / 3)
Access image generation, edits, and variations via `POST /v1/images/generations`:
```bash
curl -X POST http://localhost:8080/v1/images/generations \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "dall-e-3",
"prompt": "Isometric view of a modern data center with glowing fiber cables, cinematic lighting",
"size": "1024x1024",
"quality": "standard",
"n": 1
}'
```
GoRoute records image unit accounting in micro-dollars based on model resolution and quality tier (e.g., $0.040 for 1024x1024 standard DALL-E 3).
---
## Realtime WebSocket Protocol (`/v1/realtime`)
GoRoute supports the OpenAI **Realtime API** (`GET /v1/realtime` and `GET /openai/v1/realtime`), enabling low-latency, full-duplex speech-to-speech and text-to-speech interactions.
- **Connection Hijacking**: GoRoute leverages `nhooyr.io/websocket` to hijack the initial HTTP request and establish bidirectional WebSocket tunnels between client applications and OpenAI.
- **Upstream Session Headers**: Injects Bearer credentials and `OpenAI-Beta: realtime=v1` transparently.
- **FinOps Stream Auditing**: Monitors session frame counts and connection durations.
```javascript
import WebSocket from 'ws';
const ws = new WebSocket('ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01', {
headers: {
'Authorization': 'Bearer your-virtual-key'
}
});
ws.on('open', () => {
ws.send(JSON.stringify({
type: 'response.create',
response: {
modalities: ['text'],
instructions: 'You are a real-time conversational agent.'
}
}));
});
```
---
## Gateway Configuration
Add OpenAI deployments to `configs/config.yaml`:
```yaml
model_list:
# Primary GPT-4o
- model_name: gpt-4o
routing_strategy: weighted-shuffle
fallbacks: ["gpt-4o-mini"]
model_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
rpm: 5000
tpm: 800000
# Cost-effective fallback
- model_name: gpt-4o-mini
model_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
# Reasoning flagships
- model_name: o3-mini
model_params:
model: openai/o3-mini
api_key: os.environ/OPENAI_API_KEY
- model_name: o1
model_params:
model: openai/o1
api_key: os.environ/OPENAI_API_KEY
# Embeddings
- model_name: text-embedding-3-large
model_params:
model: openai/text-embedding-3-large
api_key: os.environ/OPENAI_API_KEY
```
---
## Usage Examples
### 1. Chat Completion with Reasoning Effort
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "o3-mini",
"reasoning_effort": "medium",
"messages": [
{"role": "user", "content": "Write a concurrent worker pool in Go with worker cancellation."}
]
}'
```
### 2. Streaming with Usage Statistics
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Summarize distributed consensus algorithms."}],
"stream": true,
"stream_options": {"include_usage": true}
}'
```
### 3. Calling OpenAI via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "gpt-4o",
Model: "openai/gpt-4o",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Gateway initialization failed: %v", err)
}
defer gw.Close()
response, err := gw.Chat(ctx, "gpt-4o", "Explain zero-copy I/O in the Linux kernel.")
if err != nil {
log.Fatalf("OpenAI chat error: %v", err)
}
fmt.Println("Response:\n", response)
}
```
> [!NOTE]
> GoRoute automatically registers upstream model pricing provenance for OpenAI models, ensuring real-time spend calculations reflect the latest token rates for prompt, completion, and reasoning tokens.
---
## Anthropic Claude
URL: https://goroute-ai.github.io/docs/providers/anthropic-claude
Markdown: https://goroute-ai.github.io/docs/providers/anthropic-claude.md
Section: Control Plane Documentation > Choose a model provider
# Anthropic Claude
GoRoute provides a native adapter for **Anthropic**, translating standard OpenAI chat completion and tool calling requests into Anthropic's **Messages API** (`POST /v1/messages`). The model IDs below are provider-side examples, not an exhaustive or guaranteed-current list of models supported by every GoRoute deployment. Check [Anthropic's current model status](https://docs.anthropic.com/en/docs/about-claude/model-deprecations) and the [model overview](https://docs.anthropic.com/en/docs/about-claude/models), then confirm which models your key can use with [`GET /v1/models`](https://goroute-ai.github.io/docs/api/gateway-api#4-model-catalogue-inventory-get-v1models).
The adapter natively implements Anthropic-specific protocol requirements (including extended thinking budgets, strict user/assistant turn alternation, tool use consolidation, and Server-Sent Events (SSE) stream translation).
---
## Representative model IDs
| Model family | Example Anthropic identifier |
| :--- | :--- |
| Claude Sonnet | `claude-sonnet-5-5` |
| Claude Opus | `claude-opus-5-5` |
| Claude Haiku | `claude-haiku-5-5` |
These examples reflect identifiers in Anthropic's provider catalog when this page was updated. Availability, lifecycle status, and feature support can vary; the gateway inventory is the check for a particular deployment and key.
---
## Alternating Turn Tool Consolidation
Anthropic's Messages API enforces a strict architectural rule: **every conversation must alternate strictly between `user` and `assistant` turns**.
In the OpenAI protocol, when a model executes parallel function calls, the client sends back separate consecutive messages with `role: "tool"`. If sent verbatim to Anthropic, this would trigger an immediate `400 invalid_request_error` (`consecutive messages with the same role are disallowed`).
```mermaid
flowchart TD
subgraph OpenAIClient["OpenAI Client Ingress"]
direction TB
M1["assistant: tool_calls [tool_1, tool_2]"]
M2["tool: result_1 (call_id: tool_1)"]
M3["tool: result_2 (call_id: tool_2)"]
end
subgraph GoRouteConsolidation["GoRoute Turn Consolidation Engine"]
M1 --> A1["assistant turn: content_blocks [tool_use_1, tool_use_2]"]
M2 & M3 --> U1["Single user turn: content_blocks [tool_result_1, tool_result_2]"]
end
subgraph AnthropicUpstream["Anthropic Messages API"]
A1 --> Upstream["Validated Strict Alternation"]
U1 --> Upstream
end
```
GoRoute's adapter (`internal/adapter/anthropic/adapter.go`) automatically intercepts consecutive `role: "tool"` entries and consolidates them into a single `user` turn containing multiple `tool_result` content blocks:
```json
{
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": "toolu_01A...",
"content": "{\"temperature\": 72, \"condition\": \"Sunny\"}"
},
{
"type": "tool_result",
"tool_use_id": "toolu_01B...",
"content": "{\"humidity\": 45}"
}
]
}
```
---
## Extended Thinking Budgets
Anthropic Claude models that support extended thinking can be configured through the normalized reasoning interface, allowing the model to perform chain-of-thought reasoning before outputting its response.
GoRoute normalizes reasoning across providers. When you supply `reasoning_effort` (`"low"`, `"medium"`, `"high"`), GoRoute automatically configures Anthropic's `thinking` block:
| `reasoning_effort` | Budget Tokens | Behavior |
| :--- | :--- | :--- |
| `"high"` | 8,192 tokens | Deep problem decomposition and exhaustive code verification |
| `"medium"` | 4,096 tokens | Balanced reasoning for analytical questions |
| `"low"` | 2,048 tokens | Targeted reasoning step verification |
### Automatic Max Tokens Elevation
Anthropic requires that `max_tokens > thinking.budget_tokens`. If a client specifies `max_tokens: 4096` with `reasoning_effort: "high"` (budget: 8192), GoRoute automatically elevates `max_tokens` to `10240` (`budget + 2048`) to ensure the request is accepted and there is sufficient room for the final response.
Furthermore, because Anthropic strictly forbids setting custom `temperature` or `top_p` when thinking is enabled, GoRoute automatically nils out those parameters before dispatch.
---
## System Prompt Extraction
Anthropic does not allow `role: "system"` messages in its `messages` array. GoRoute automatically filters all messages with `role: "system"` or `role: "developer"`, concatenates their text with double newlines (`\n\n`), and places the result in the top-level Anthropic `system` property.
---
## Gateway Configuration
Add Anthropic models to `configs/config.yaml`:
```yaml
model_list:
# Primary: Claude Sonnet 5.5
- model_name: claude-sonnet-5-5
routing_strategy: least-busy
fallbacks: ["claude-haiku-5-5", "gpt-4o"]
model_params:
model: anthropic/claude-sonnet-5-5
api_key: os.environ/ANTHROPIC_API_KEY
rpm: 2000
tpm: 400000
# High-throughput: Claude Haiku 5.5
- model_name: claude-haiku-5-5
model_params:
model: anthropic/claude-haiku-5-5
api_key: os.environ/ANTHROPIC_API_KEY
# Deep Reasoning: Claude Opus 5.5
- model_name: claude-opus-5-5
model_params:
model: anthropic/claude-opus-5-5
api_key: os.environ/ANTHROPIC_API_KEY
```
Set the API key in your environment:
```bash
export ANTHROPIC_API_KEY="sk-ant-api03-..."
```
---
## Usage Examples
### 1. Chat Completion with Thinking Enabled
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-5-5",
"reasoning_effort": "high",
"messages": [
{"role": "system", "content": "You are a distributed systems architect."},
{"role": "user", "content": "Formulate a formal proof that Raft leader election guarantees election safety."}
]
}'
```
### 2. Streaming Chat with Tool Calls
GoRoute parses Anthropic's Server-Sent Events stream (`content_block_start`, `content_block_delta`, `message_delta`, `message_stop`) and emits standard OpenAI SSE chunks:
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-sonnet-5-5",
"messages": [{"role": "user", "content": "Explain how epoll scales differently than select."}],
"stream": true
}'
```
### 3. Calling Anthropic via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "claude-chat",
Model: "anthropic/claude-sonnet-5-5",
APIKey: "os.environ/ANTHROPIC_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Failed to initialize gateway: %v", err)
}
defer gw.Close()
response, err := gw.Chat(ctx, "claude-chat", "What are the trade-offs of LSM-trees vs B-Trees in write-heavy workloads?")
if err != nil {
log.Fatalf("Claude chat error: %v", err)
}
fmt.Println("Claude Response:\n", response)
}
```
> [!TIP]
> Anthropic Messages ingress is also available on direct Anthropic routes: `POST /v1/messages` and `POST /anthropic/v1/messages`. Anthropic SDK callers can point their base URL directly to GoRoute without modifying request code.
---
## Google Gemini
URL: https://goroute-ai.github.io/docs/providers/google-gemini
Markdown: https://goroute-ai.github.io/docs/providers/google-gemini.md
Section: Control Plane Documentation > Choose a model provider
# Google Gemini
GoRoute provides a native provider adapter for **Google Gemini** (`internal/adapter/gemini/`), translating standard OpenAI chat completion, streaming, embeddings, and tool calling requests into Google's **Gemini REST API** (`generateContent`, `streamGenerateContent`).
The adapter supports Google AI Studio API keys, multimodal audio and image inputs, structured JSON outputs, image generation, and reasoning-budget handling. The model IDs shown here are provider-side examples, not an exhaustive or guaranteed-current list of models supported by every GoRoute deployment. Check [Google's current Gemini model list](https://ai.google.dev/gemini-api/docs/models), then confirm which models your key can use with [`GET /v1/models`](https://goroute-ai.github.io/docs/api/gateway-api#4-model-catalogue-inventory-get-v1models).
---
## Representative model IDs
| Model family / task | Example Gemini API identifier |
| :--- | :--- |
| Gemini Flash | `gemini-3.8-flash` |
| Gemini Flash (previous generation) | `gemini-2.5-flash` |
| Text embeddings | `gemini-embedding-001` |
| Image generation | `gemini-3.1-flash-image` |
These examples refer to the Gemini API catalog. Google AI Studio and Vertex AI can expose different model IDs and availability. An ID listed by Google does not confirm that it is enabled for your GoRoute deployment or compatible with every gateway route; check the gateway inventory and the relevant Google endpoint documentation.
---
## Thinking Budget Controls (Gemini 2.5 Flash)
Gemini 2.5 Flash includes integrated chain-of-thought thinking capabilities. However, when a client requests a very short completion (for example, classification or sentiment analysis with `max_tokens <= 16`), an unconstrained thinking process can consume the entire token budget on internal thoughts, resulting in an empty response string.
GoRoute implements an automated guardrail in `internal/adapter/gemini/adapter.go`:
```go
// When max_tokens is small (<= 16) on Gemini 2.5 Flash without explicit reasoning effort,
// set thinkingBudget: 0 so all output tokens go to visible response text rather than internal thought tokens.
if req.MaxTokens != nil && *req.MaxTokens <= 16 && req.ReasoningEffort == "" && model == "gemini-2.5-flash" {
genCfg.ThinkingConfig = &providers.GeminiThinkingConfig{ThinkingBudget: 0}
}
```
This guarantees that compact responses (such as "YES", "NO", or category codes) are immediately returned without being starved by internal thought generation.
When deeper reasoning is needed, specify `reasoning_effort` (`"low"`, `"medium"`, or `"high"`) in your request to configure proportional token budgets.
---
## Multimodal Audio & Vision Ingress
Gemini models feature native multimodal processing. Clients can pass high-resolution images or audio files directly within the OpenAI message content array:
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-2.5-flash",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the architecture diagram in this image:"},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
}
}
]
}
]
}'
```
---
## Schema Translation: OpenAI to Gemini
The Gemini adapter executes bidirectional transformation between OpenAI REST schemas and Gemini's content object model:
| OpenAI Schema | Gemini REST Mapping |
| :--- | :--- |
| `role: "system"` or `"developer"` | Top-level `systemInstruction` with `role: "user"` |
| `role: "assistant"` | `role: "model"` |
| `role: "user"` | `role: "user"` |
| `role: "tool"` | `role: "function"` with `functionResponse` |
| `tools[].function` | `tools[].functionDeclarations[]` |
| `response_format: {"type": "json_object"}` | `generationConfig.responseMimeType: "application/json"` |
| `response_format: {"type": "json_schema"}` | `generationConfig.responseSchema` |
---
## Gateway Configuration
Add Gemini deployments to `configs/config.yaml`:
```yaml
model_list:
# Primary: Gemini Flash example; verify this ID in your gateway inventory first
- model_name: gemini-fast
routing_strategy: weighted-shuffle
fallbacks: ["gemini-pro"]
model_params:
model: gemini/gemini-3.8-flash
api_key: os.environ/GEMINI_API_KEY
rpm: 1000
tpm: 1000000
# Previous-generation Pro example
- model_name: gemini-pro
model_params:
model: gemini/gemini-2.5-pro
api_key: os.environ/GEMINI_API_KEY
# Gemini Embeddings
- model_name: gemini-embedding
model_params:
model: gemini/gemini-embedding-001
api_key: os.environ/GEMINI_API_KEY
```
Set the API key in your environment:
```bash
export GEMINI_API_KEY="AIzaSy..."
```
---
## Usage Examples
### 1. Chat Completion via cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-fast",
"messages": [
{"role": "system", "content": "You are a concise technical writer."},
{"role": "user", "content": "What is the function of an L4 load balancer?"}
],
"temperature": 0.2,
"max_tokens": 300
}'
```
### 2. Streaming Responses
The Gemini adapter parses upstream Server-Sent Events (`streamGenerateContent?alt=sse`) and converts candidate content chunks into standard OpenAI SSE frames:
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-fast",
"messages": [{"role": "user", "content": "Explain how Go channels are implemented."}],
"stream": true
}'
```
### 3. Calling Gemini via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "gemini-fast",
Model: "gemini/gemini-2.5-flash",
APIKey: "os.environ/GEMINI_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Gateway initialization failed: %v", err)
}
defer gw.Close()
response, err := gw.Chat(ctx, "gemini-fast", "Summarize why vector embeddings are critical for RAG systems.")
if err != nil {
log.Fatalf("Gemini chat error: %v", err)
}
fmt.Println("Gemini Reply:\n", response)
}
```
> [!TIP]
> If your enterprise agreement requires routing Gemini traffic through Google Cloud IAM, VPC Service Controls, or specific European regions, use the native [Google Cloud Vertex AI adapter](https://goroute-ai.github.io/docs/providers/google-vertex) instead of the Gemini API Studio adapter.
---
## AWS Bedrock
URL: https://goroute-ai.github.io/docs/providers/aws-bedrock
Markdown: https://goroute-ai.github.io/docs/providers/aws-bedrock.md
Section: Control Plane Documentation > Choose a model provider
# AWS Bedrock
GoRoute provides a native adapter for Amazon Web Services (AWS) Bedrock (`internal/adapter/bedrock/`), enabling unified access to foundation models through AWS's runtime APIs. Model families, IDs, and regional availability change; consult AWS's current catalog and the representative-ID guidance below.
Rather than relying on legacy invocation endpoints, GoRoute leverages the AWS Bedrock **Converse API** (`/model/{model}/converse` and `/model/{model}/converse-stream`). This guarantees uniform multi-turn message structures, standardized tool use schemas, native system prompt injection, and binary AWS EventStream chunk decoding.
---
## Architecture & Authentication
AWS Bedrock requires cryptographic request signing via **AWS Signature Version 4 (SigV4)**. Every HTTP request dispatched to the Bedrock runtime includes canonical header hashes, HMAC-SHA256 signatures, and timestamps.
```mermaid
flowchart LR
Client["Client / SDK"] -->|Standard OpenAI Request| GW["GoRoute AI Gateway"]
GW --> SigV4["AWS SigV4 Signer\n(internal/adapter/bedrock/sigv4.go)"]
SigV4 -->|HMAC-SHA256 Auth\nBedrock Converse API| Bedrock["AWS Bedrock Runtime\n(bedrock-runtime.{region}.amazonaws.com)"]
Bedrock -->|Binary EventStream| GW
GW -->|Standard OpenAI SSE| Client
```
### Authentication Strategies
GoRoute supports three flexible authentication methods for AWS Bedrock credentials:
#### 1. Structured JSON Credential (Recommended)
Store the full IAM configuration within a single encrypted credential string or environment variable:
```json
{
"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"aws_region": "us-east-1",
"aws_session_token": "AQoDYXdzEJr1...EXAMPLE"
}
```
> [!NOTE]
> `aws_session_token` is optional and only required when authenticating with temporary credentials from AWS STS (e.g., cross-account IAM role assumption).
#### 2. Colon-Delimited Credential
Pass access keys using the standard colon-separated format:
- `access_key_id:secret_access_key`
- `access_key_id:secret_access_key:region`
- `access_key_id:secret_access_key:region:session_token`
#### 3. Ambient Environment Variables
If only the Access Key ID is supplied in the credential, GoRoute falls back to ambient AWS environment variables:
- `AWS_ACCESS_KEY_ID`
- `AWS_SECRET_ACCESS_KEY`
- `AWS_REGION` (defaults to `us-east-1` if unset)
- `AWS_SESSION_TOKEN` (optional)
---
## Representative model IDs
This table is illustrative and may lag behind AWS releases. See [AWS Bedrock's current supported model list](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) for provider-side IDs, then check [`GET /v1/models`](https://goroute-ai.github.io/docs/api/gateway-api#4-model-catalogue-inventory-get-v1models) to confirm which models are exposed to your gateway key. AWS account access, region, gateway catalog state, and model policies affect availability.
| Foundation Family | Example Bedrock Model ID |
| :--- | :--- |
| **Claude Haiku 5.5** | `anthropic.claude-haiku-5-5` |
| **Claude Sonnet 5.5** | `anthropic.claude-sonnet-5-5` |
| **Claude Opus 5.5** | `anthropic.claude-opus-5-5` |
| **Other available model families** | See AWS's live model catalog for current IDs, regions, modalities, and endpoint support. |
> [!IMPORTANT]
> Make sure your AWS Account has requested and been granted model access in the AWS Bedrock Console for the specific region you target (e.g., `us-east-1` or `us-west-2`).
---
## Gateway Configuration
Add your Bedrock deployments to `configs/config.yaml`:
```yaml
model_list:
# Claude Sonnet 5.5 on AWS Bedrock; confirm region and account access
- model_name: bedrock-claude
model_params:
model: bedrock/anthropic.claude-sonnet-5-5
api_key: os.environ/AWS_BEDROCK_JSON_CREDENTIALS
# Alternatively, specify custom endpoint or VPC endpoint:
# api_base: "https://vpce-0123456789abcdef0.bedrock-runtime.us-east-1.vpce.amazonaws.com"
# Replace this placeholder with an available Bedrock model ID
- model_name: bedrock-llama
model_params:
model: bedrock/
api_key: os.environ/AWS_BEDROCK_KEY_PAIR # e.g. "AKIA...:SECRET..."
# Replace this placeholder with an available Bedrock embedding model ID
- model_name: bedrock-titan-embed
model_params:
model: bedrock/
api_key: os.environ/AWS_BEDROCK_JSON_CREDENTIALS
```
### Environment Configuration
Export the credentials in your environment:
```bash
# JSON Credential option
export AWS_BEDROCK_JSON_CREDENTIALS='{"aws_access_key_id":"AKIA...","aws_secret_access_key":"...","aws_region":"us-east-1"}'
# Or standard AWS environment variables
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="us-east-1"
```
---
## Usage Examples
### 1. Chat Completion via cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "bedrock-claude",
"messages": [
{"role": "system", "content": "You are a cloud architecture assistant."},
{"role": "user", "content": "Explain AWS VPC peering vs Transit Gateway in 3 bullet points."}
],
"temperature": 0.2,
"max_tokens": 500
}'
```
### 2. Streaming with EventStream Decoding
Bedrock delivers streaming tokens over a binary AWS EventStream protocol. GoRoute automatically parses binary EventStream frames, handles exception blocks, and maps `contentBlockDelta` events into standard OpenAI Server-Sent Events (SSE):
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "bedrock-claude",
"messages": [{"role": "user", "content": "Write a Go function to verify an AWS SigV4 signature."}],
"stream": true
}'
```
```text
data: {"id":"bedrock-1731336492","object":"chat.completion.chunk","created":1731336492,"model":"anthropic.claude-sonnet-5-5","choices":[{"index":0,"delta":{"content":"package"},"finish_reason":null}]}
...
data: {"id":"bedrock-1731336492","object":"chat.completion.chunk","created":1731336492,"model":"anthropic.claude-sonnet-5-5","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]
```
### 3. Embeddings via Amazon Bedrock
Embedding models available to your AWS account can be invoked through the unified `/v1/embeddings` endpoint. Use an ID from the current Bedrock catalog and gateway inventory:
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "bedrock-titan-embed",
"input": ["Enterprise Retrieval-Augmented Generation at scale."]
}'
```
### 4. Calling Bedrock via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "bedrock-claude",
Model: "bedrock/anthropic.claude-sonnet-5-5",
APIKey: "os.environ/AWS_BEDROCK_JSON_CREDENTIALS",
},
),
)
if err != nil {
log.Fatalf("Failed to initialize gateway: %v", err)
}
defer gw.Close()
resp, err := gw.Chat(ctx, "bedrock-claude", "List three benefits of running LLMs in your own VPC.")
if err != nil {
log.Fatalf("Chat error: %v", err)
}
fmt.Println("Response:\n", resp)
}
```
---
## Tool Calling & Structured Outputs
The Bedrock adapter maps OpenAI function tool specifications directly to Bedrock Converse tool specifications:
```json
{
"toolConfig": {
"tools": [
{
"toolSpec": {
"name": "lookup_customer",
"description": "Retrieve customer subscription data",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"customer_id": {"type": "string"}
},
"required": ["customer_id"]
}
}
}
}
]
}
}
```
When the upstream model initiates a tool call, GoRoute converts Bedrock's `toolUse` block into an OpenAI-compatible tool call with an ID and JSON string arguments.
> [!TIP]
> When integrating AWS Bedrock with VPC endpoints (`com.amazonaws.{region}.bedrock-runtime`), specify the private endpoint in `model_params.api_base`. GoRoute's SSRF policy validates the custom URL against allowed private CIDRs.
---
## Google Cloud Vertex AI
URL: https://goroute-ai.github.io/docs/providers/google-vertex
Markdown: https://goroute-ai.github.io/docs/providers/google-vertex.md
Section: Control Plane Documentation > Choose a model provider
# Google Cloud Vertex AI
GoRoute provides a native enterprise adapter for **Google Cloud Vertex AI** (`internal/adapter/vertex/`), enabling organizations to consume Google's Gemini models through their existing Google Cloud Platform (GCP) enterprise agreements, VPC Service Controls, and IAM role hierarchies.
Model availability and IDs on Vertex AI can differ from Google AI Studio. The model identifiers in the examples below are illustrative, not a complete or guaranteed-current support list. Check [Vertex AI's model documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models) and confirm the models exposed to your caller with [`GET /v1/models`](https://goroute-ai.github.io/docs/api/gateway-api#4-model-catalogue-inventory-get-v1models).
The Vertex AI adapter directly communicates with the Vertex AI Platform REST endpoints:
```text
https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/publishers/google/models/{model}:generateContent
```
---
## Authentication & Token Exchange
Vertex AI requires an **OAuth2 Bearer access token** scoped to `https://www.googleapis.com/auth/cloud-platform`. GoRoute implements a built-in cryptographic token exchange pipeline (`internal/adapter/vertex/auth.go`) that eliminates the need for sidecars or local `gcloud` installations.
```mermaid
sequenceDiagram
participant GW as GoRoute AI Gateway
participant OAuth as Google OAuth2
participant Vertex as Vertex AI Endpoint
GW->>GW: 1. Load GCP Service Account JSON
GW->>OAuth: 2. POST /token (Signed Assertion JWT)
OAuth-->>GW: 3. Access Token (Bearer ya29... expires 3600s)
GW->>GW: 4. Cache Token (sync.RWMutex)
GW->>Vertex: 5. POST :generateContent (Bearer ya29...)
Vertex-->>GW: 6. Model Response / SSE Stream
```
### Supported Credential Formats
#### 1. GCP Service Account JSON Key (Recommended)
Supply your complete Google Cloud Service Account JSON key as the credential value (or point to an environment variable containing it):
```json
{
"type": "service_account",
"project_id": "production-data-platform",
"private_key_id": "b1a2c3d4e5f6...",
"private_key": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEA0...\n-----END RSA PRIVATE KEY-----\n",
"client_email": "ai-gateway@production-data-platform.iam.gserviceaccount.com",
"client_id": "109876543210987654321",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/..."
}
```
GoRoute:
1. Parses either PKCS#8 or PKCS#1 RSA private keys.
2. Constructs a signed RFC 7515 RS256 JWT assertion with claims (`iss`, `scope`, `aud`, `exp`, `iat`).
3. Dispatches a grant type request (`urn:ietf:params:oauth:grant-type:jwt-bearer`) to `https://oauth2.googleapis.com/token`.
4. Caches the token with thread-safe `sync.RWMutex` locking and automatically refreshes it prior to expiry.
#### 2. Direct Bearer Token
If your workload already runs within Google Kubernetes Engine (GKE) with Workload Identity or Google Compute Engine with metadata tokens, pass the raw Bearer token directly (`ya29.c...`).
---
## Regional Configuration & Endpoints
Vertex AI is regionally distributed. By default, GoRoute routes to `us-central1`. You can customize the region and GCP project ID via configuration or environment variables:
| Setting | Configuration Field | Environment Variable | Default |
| :--- | :--- | :--- | :--- |
| **GCP Region** | Derived from `api_base` or env | `VERTEX_LOCATION` or `GCP_REGION` | `us-central1` |
| **GCP Project** | Extracted from SA JSON or env | `VERTEX_PROJECT_ID` or `GCP_PROJECT_ID` | SA `project_id` |
| **Base URL** | `model_params.api_base` | `VERTEX_BASE_URL` | `https://{region}-aiplatform.googleapis.com` |
Supported regional endpoints include `us-central1`, `us-east4`, `us-west1`, `europe-west1`, `europe-west4`, `asia-northeast1`, and more.
---
## Model availability
Vertex AI publishes model IDs and regional availability in its [current model catalog](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models). Use that catalog to select a provider-side ID, then check the gateway inventory to confirm it is exposed to your key. The model ID placeholder in the configuration below is intentional: Vertex AI IDs can differ from Google AI Studio and change over time.
---
## Gateway Configuration
Add your Vertex AI deployments to `configs/config.yaml`:
```yaml
model_list:
# Replace the placeholder with a current Vertex AI model ID exposed by your gateway
- model_name: vertex-gemini-flash
model_params:
model: vertex/
api_key: os.environ/GCP_VERTEX_SA_KEY
# Optional explicit base URL override:
# api_base: "https://us-central1-aiplatform.googleapis.com"
# The provider model ID and region must be available to your Vertex AI project
- model_name: vertex-gemini-pro-eu
model_params:
model: vertex/
api_key: os.environ/GCP_VERTEX_SA_KEY
api_base: "https://europe-west4-aiplatform.googleapis.com"
# Vertex AI Text Embeddings
- model_name: vertex-embed
model_params:
model: vertex/
api_key: os.environ/GCP_VERTEX_SA_KEY
```
Set the Service Account JSON in your environment:
```bash
export GCP_VERTEX_SA_KEY=$(cat /path/to/vertex-sa-key.json)
export VERTEX_LOCATION="us-central1"
export VERTEX_PROJECT_ID="production-data-platform"
```
---
## Usage Examples
### 1. Chat Completion via cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-gemini-flash",
"messages": [
{"role": "system", "content": "You are an enterprise GCP security consultant."},
{"role": "user", "content": "How do VPC Service Controls mitigate data exfiltration in BigQuery?"}
],
"temperature": 0.2,
"max_tokens": 1000
}'
```
### 2. Streaming Chat with Server-Sent Events
The Vertex AI adapter streams chunks using Vertex AI's `:streamGenerateContent?alt=sse` endpoint and translates candidate parts into standard OpenAI SSE chunks:
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-gemini-flash",
"messages": [{"role": "user", "content": "Explain Kubernetes pod disruption budgets."}],
"stream": true
}'
```
### 3. Embeddings via Vertex AI
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "vertex-embed",
"input": ["Enterprise governance and model routing with GoRoute."]
}'
```
### 4. Go SDK Integration
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "vertex-gemini",
Model: "vertex/gemini-2.5-flash",
APIKey: "os.environ/GCP_VERTEX_SA_KEY",
},
),
)
if err != nil {
log.Fatalf("Gateway initialization failed: %v", err)
}
defer gw.Close()
reply, err := gw.Chat(ctx, "vertex-gemini", "What are the three pillars of zero-trust architecture?")
if err != nil {
log.Fatalf("Vertex chat error: %v", err)
}
fmt.Println("Gemini Reply:\n", reply)
}
```
---
## Role Translation & Tool Calling
The Vertex AI adapter handles all dialect translations between OpenAI's message format and Google's Content schema:
- **System Messages**: OpenAI `role: "system"` or `role: "developer"` is extracted and sent as Vertex's top-level `systemInstruction` with `role: "user"`.
- **Assistant Messages**: OpenAI `role: "assistant"` maps to Vertex `role: "model"`.
- **Tool Invocations**: OpenAI tool calls map to `functionCall` objects; tool return values map to `role: "function"` with `functionResponse`.
- **Structured Outputs**: Specifying `response_format: {"type": "json_object"}` sets `responseMimeType: "application/json"` in the Vertex generation config.
> [!TIP]
> Ensure the Service Account has the **Vertex AI User** (`roles/aiplatform.user`) IAM role on your GCP project. If you are using custom regional endpoints, also ensure network egress allows outbound connections to `*-aiplatform.googleapis.com:443`.
---
## Azure OpenAI Service
URL: https://goroute-ai.github.io/docs/providers/azure-openai
Markdown: https://goroute-ai.github.io/docs/providers/azure-openai.md
Section: Control Plane Documentation > Choose a model provider
# Azure OpenAI Service
GoRoute provides comprehensive support for Microsoft **Azure OpenAI Service** through its dedicated Azure adapter strategy (`internal/adapter/azure/`). Organizations with dedicated Azure enterprise commitments, compliance boundaries, and private network endpoints can integrate Azure OpenAI models seamlessly into the GoRoute routing topology.
---
## Architecture & Endpoint Modes
Azure OpenAI Service supports two distinct routing architectures. GoRoute automatically inspects your endpoint configuration to select the optimal strategy:
```mermaid
flowchart TD
Client["Client / Application"] -->|Standard OpenAI POST /v1/chat/completions| GW["GoRoute AI Gateway"]
GW --> ModeCheck{"Check Base URL & Headers"}
ModeCheck -->|BaseURL ends with /openai/v1 or /v1| StandardV1["Modern Standard v1 Mode\n{base_url}/openai/v1/chat/completions"]
ModeCheck -->|Legacy deployment path| Legacy["Legacy Deployment Mode\n{base_url}/openai/deployments/{azure_deployment_name}/chat/completions?api-version={api_version}"]
StandardV1 --> Azure["Azure OpenAI Resource Endpoint"]
Legacy --> Azure
```
### 1. Modern Standard v1 Mode (`AzureEndpointModeStandardV1`)
Used when `api_base` ends in `/openai/v1` or `/v1`. This mode matches the standard OpenAI API layout while maintaining Azure authentication:
```text
https://my-resource.openai.azure.com/openai/v1/chat/completions
https://my-resource.openai.azure.com/openai/v1/embeddings
https://my-resource.openai.azure.com/openai/v1/models
```
### 2. Legacy Deployment Mode (`AzureEndpointModeLegacyDeployment`)
Used when targeting specific Azure deployment names and pinned API versions:
```text
https://my-resource.openai.azure.com/openai/deployments/{azure_deployment_name}/chat/completions?api-version=2024-08-01-preview
```
---
## Authentication Strategies
GoRoute's `AzureAuthStrategy` (`internal/adapter/azure/strategies.go`) supports both primary authentication schemes:
### 1. Azure API Key Auth
Pass your standard 32-character Azure resource API key. GoRoute injects the required `api-key` header:
```http
api-key: 7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d
```
### 2. Microsoft Entra ID (Azure AD) OAuth2 Token Auth
For zero-trust environments where shared API keys are prohibited, supply a Microsoft Entra ID Bearer token (prefixed with `Bearer `). GoRoute automatically detects the prefix and sets the standard Authorization header:
```http
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIs...
```
> [!TIP]
> You can acquire Entra ID tokens via Azure Managed Identity, Azure CLI (`az account get-access-token --resource https://cognitiveservices.azure.com`), or your enterprise token broker.
---
## Deployment Parameters & Version Pinning
In Azure OpenAI, models are accessed via customized **Deployment Names** rather than canonical model IDs. GoRoute maps these parameters cleanly:
| Parameter | YAML Configuration | Description | Example |
| :--- | :--- | :--- | :--- |
| **Endpoint Base** | `api_base` | The root HTTPS endpoint of your Azure OpenAI resource | `https://my-enterprise.openai.azure.com` |
| **Deployment Name** | `azure_deployment_name` | The custom name chosen during deployment in Azure AI Foundry | `gpt-4o-prod-eastus` |
| **API Version** | `api_version` | The pinned Azure REST API version | `2024-08-01-preview` or `2024-10-21` |
| **Endpoint Mode** | `headers.azure_endpoint_mode` | Optional override: `"v1"` or `"legacy_deployment"` | `legacy_deployment` |
GoRoute also automatically handles Azure parameter translation:
- Translates legacy `max_tokens` to `max_completion_tokens` when required.
- Preserves all custom enterprise headers while filtering out control directives (`api-version`, `azure_deployment_name`).
---
## Gateway Configuration
Configure Azure OpenAI deployments in `configs/config.yaml`:
```yaml
model_list:
# Primary: Production Azure OpenAI GPT-4o
- model_name: enterprise-gpt4o
routing_strategy: least-busy
fallbacks: ["azure-gpt4o-dr", "openai-direct-fallback"]
model_params:
model: azure/gpt-4o
api_key: os.environ/AZURE_OPENAI_API_KEY
api_base: "https://enterprise-eastus.openai.azure.com"
azure_deployment_name: "gpt-4o-prod"
api_version: "2024-08-01-preview"
headers:
X-Enterprise-Cost-Center: "CC-90210"
# Disaster Recovery: Secondary Azure Region (West US 3)
- model_name: azure-gpt4o-dr
model_params:
model: azure/gpt-4o
api_key: os.environ/AZURE_OPENAI_WESTUS_KEY
api_base: "https://enterprise-westus3.openai.azure.com"
azure_deployment_name: "gpt-4o-westus-backup"
api_version: "2024-08-01-preview"
# Entra ID Authenticated Deployment
- model_name: azure-entraid-chat
model_params:
model: azure/gpt-4o-mini
api_key: os.environ/AZURE_ENTRA_BEARER_TOKEN # "Bearer eyJhbGci..."
api_base: "https://enterprise-gov.openai.azure.com"
azure_deployment_name: "gpt-4o-mini-secure"
api_version: "2024-10-21"
# Azure Embeddings
- model_name: azure-embeddings
model_params:
model: azure/text-embedding-3-large
api_key: os.environ/AZURE_OPENAI_API_KEY
api_base: "https://enterprise-eastus.openai.azure.com"
azure_deployment_name: "text-embed-3-large"
api_version: "2024-08-01-preview"
```
---
## Usage Examples
### 1. Chat Completion via cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "enterprise-gpt4o",
"messages": [
{"role": "system", "content": "You are a corporate financial analyst."},
{"role": "user", "content": "Summarize Q3 operating margin trends."}
],
"temperature": 0.3,
"max_tokens": 600
}'
```
### 2. Streaming Responses
GoRoute parses Azure's Server-Sent Events stream, ensuring identical response shape to direct OpenAI endpoints:
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "enterprise-gpt4o",
"messages": [{"role": "user", "content": "Draft an executive summary for a cloud migration RFP."}],
"stream": true
}'
```
### 3. Embeddings via Azure OpenAI
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "azure-embeddings",
"input": ["Azure Private Link endpoint connection validation."]
}'
```
### 4. Go SDK Integration
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "azure-chat",
Model: "azure/gpt-4o",
APIKey: "os.environ/AZURE_OPENAI_API_KEY",
APIBase: "https://enterprise-eastus.openai.azure.com",
},
),
)
if err != nil {
log.Fatalf("Gateway init failed: %v", err)
}
defer gw.Close()
response, err := gw.Chat(ctx, "azure-chat", "What are the core components of Azure Landing Zones?")
if err != nil {
log.Fatalf("Azure chat failed: %v", err)
}
fmt.Println("Azure OpenAI Response:\n", response)
}
```
> [!IMPORTANT]
> If your Azure OpenAI Service is locked down via **Azure Private Endpoints (Private Link)**, configure GoRoute's network egress in `configs/config.yaml` to allow private CIDRs (`security.network.outbound_ssrf.allow_private_networks: true` and specify `allowed_private_cidrs: ["10.0.0.0/16"]`).
---
## Cohere
URL: https://goroute-ai.github.io/docs/providers/cohere
Markdown: https://goroute-ai.github.io/docs/providers/cohere.md
Section: Control Plane Documentation > Choose a model provider
# Cohere
GoRoute provides a native provider adapter for **Cohere** (`internal/adapter/cohere/`), delivering full enterprise support for Cohere's **v2 API** suite. This includes conversational generation with **Command R** and **Command R+**, multilingual dense vector embeddings with **Cohere Embed v3**, state-of-the-art document reranking with **Cohere Rerank v3.5**, and exact micro-dollar accounting for Rerank search units.
---
## Supported Modalities & Endpoints
| Modality | Gateway Route | Upstream Cohere Endpoint | Primary Models |
| :--- | :--- | :--- | :--- |
| **Chat & Tools** | `POST /v1/chat/completions` | `POST /v2/chat` | `command-r-plus-08-2024`, `command-r-08-2024` |
| **Embeddings** | `POST /v1/embeddings` | `POST /v2/embed` | `embed-english-v3.0`, `embed-multilingual-v3.0` |
| **Document Rerank** | `POST /v1/rerank`, `/rerank`, `/v2/rerank` | `POST /v2/rerank` | `rerank-v3.5`, `rerank-english-v3.0` |
---
## High-Precision Document Reranking
Cohere's Rerank API dramatically improves the accuracy of Retrieval-Augmented Generation (RAG) pipelines by scoring candidate documents against a search query using cross-encoders.
GoRoute implements native rerank handling in `internal/server/handlers_rerank.go` and `internal/adapter/cohere/adapter.go`:
```mermaid
sequenceDiagram
participant App as RAG Application
participant GW as GoRoute AI Gateway
participant Cohere as Cohere API (/v2/rerank)
App->>GW: 1. POST /v1/rerank (query, documents, top_n)
GW->>Cohere: 2. POST /v2/rerank (Bearer Key)
Cohere-->>GW: 3. Results (relevance scores & billed units)
Note over GW: FinOps Ledger records micro-dollars
GW-->>App: 4. HTTP 200 OK (Cost Header + Results)
```
### Request & Response Schema
#### Request (`POST /v1/rerank`, `POST /rerank`, or `POST /v2/rerank`)
```json
{
"model": "rerank-v3.5",
"query": "What are the latency advantages of wafer-scale engines?",
"documents": [
"Wafer-scale engines integrate an entire wafer onto a single chip, cutting off-chip interconnect bottlenecks.",
"Traditional GPU clusters rely on InfiniBand networks with microsecond scale hops.",
"Solar panel production utilizes silicon wafers cut from cylindrical ingots."
],
"top_n": 2,
"return_documents": true,
"max_tokens_per_doc": 4096
}
```
#### Response
```json
{
"id": "rerank-080b06b9-e137-4f6c-859b-14d23253b27b",
"results": [
{
"index": 0,
"relevance_score": 0.9842,
"document": {
"text": "Wafer-scale engines integrate an entire wafer onto a single chip, cutting off-chip interconnect bottlenecks."
}
},
{
"index": 1,
"relevance_score": 0.6518,
"document": {
"text": "Traditional GPU clusters rely on InfiniBand networks with microsecond scale hops."
}
}
],
"meta": {
"billed_units": {
"search_units": 1
}
}
}
```
---
## FinOps Search Unit Accounting
Unlike token-based generation models, Cohere Rerank is billed per **Search Unit** (typically 1 search unit per query and up to 100 documents, subject to token length).
GoRoute tracks rerank usage directly in the FinOps ledger (`internal/finops/calculator.go`):
- Reads `meta.billed_units.search_units` from the upstream response.
- Multiplies by the configured model rate (`RerankCostMicrosPerSearchUnit`).
- Emits cost telemetry in the `X-Gateway-Cost-Micros` response header.
- Enforces real-time organization spending quotas before request dispatch.
> [!NOTE]
> 1 Search Unit at Cohere standard pricing is $0.002 (2,000 micro-dollars). GoRoute accounts for this with 64-bit integer micro-dollar precision without floating-point drift.
---
## Gateway Configuration
Add Cohere models to `configs/config.yaml`:
```yaml
model_list:
# Cohere Command R+ (Chat & Tools)
- model_name: cohere-command-r-plus
model_params:
model: cohere/command-r-plus-08-2024
api_key: os.environ/COHERE_API_KEY
# Cohere Command R (Fast Chat)
- model_name: cohere-command-r
model_params:
model: cohere/command-r-08-2024
api_key: os.environ/COHERE_API_KEY
# Cohere Embed English v3
- model_name: cohere-embed
model_params:
model: cohere/embed-english-v3.0
api_key: os.environ/COHERE_API_KEY
# Cohere Rerank v3.5
- model_name: rerank-v3.5
model_params:
model: cohere/rerank-v3.5
api_key: os.environ/COHERE_API_KEY
```
Set your Cohere API key:
```bash
export COHERE_API_KEY="your-cohere-api-key"
```
---
## Usage Examples
### 1. Document Reranking via cURL
```bash
curl -X POST http://localhost:8080/v1/rerank \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "rerank-v3.5",
"query": "How does vector database index caching work?",
"documents": [
"Vector indexes like HNSW build proximity graphs in memory to accelerate nearest neighbor queries.",
"Relational databases use B-Trees for structured range indexing.",
"Redis is an in-memory key-value cache used for session persistence."
],
"top_n": 1,
"return_documents": true
}'
```
Response includes telemetry headers:
```http
HTTP/1.1 200 OK
X-Gateway-Provider: cohere
X-Gateway-Model: rerank-v3.5
X-Gateway-Cost-Micros: 2000
Content-Type: application/json
```
### 2. Conversational Generation (Command R+)
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere-command-r-plus",
"messages": [
{"role": "user", "content": "What is the difference between dense and sparse embeddings?"}
],
"temperature": 0.3
}'
```
### 3. Embeddings via Cohere Embed v3
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "cohere-embed",
"input": ["Cohere Embed v3 produces high-quality document embeddings."]
}'
```
### 4. Calling Cohere via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "cohere-chat",
Model: "cohere/command-r-plus-08-2024",
APIKey: "os.environ/COHERE_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Gateway initialization failed: %v", err)
}
defer gw.Close()
reply, err := gw.Chat(ctx, "cohere-chat", "Explain the concept of semantic search.")
if err != nil {
log.Fatalf("Chat failed: %v", err)
}
fmt.Println("Cohere Command R+ Reply:\n", reply)
}
```
> [!TIP]
> Cohere supports structured documents in rerank requests. You can pass objects with `rank_fields` (e.g. `["title", "body"]`) to evaluate multi-field records accurately without string flattening.
---
## Mistral AI
URL: https://goroute-ai.github.io/docs/providers/mistral
Markdown: https://goroute-ai.github.io/docs/providers/mistral.md
Section: Control Plane Documentation > Choose a model provider
# Mistral AI
GoRoute provides a native adapter for **Mistral AI** (`internal/adapter/mistral/`), providing unified access to European frontier models including **Mistral Large**, **Codestral** (dedicated coding model), **Pixtral** (multimodal vision), and **Mistral Embeddings**.
The adapter handles complete protocol translation, streaming SSE token accumulation, automatic model discovery, and tool calling normalization.
---
## Representative model IDs
This list is illustrative rather than exhaustive. Mistral model names can change; see [Mistral's current model documentation](https://docs.mistral.ai/getting-started/models/models_overview/) and confirm which IDs your gateway key can use with [`GET /v1/models`](https://goroute-ai.github.io/docs/api/gateway-api#4-model-catalogue-inventory-get-v1models). Catalog discovery is configuration-dependent and does not override provider access or gateway policies.
| Model | Identifier | Modality | Max Context | Description |
| :--- | :--- | :--- | :--- | :--- |
| **Mistral Large** | `mistral-large-latest` | Chat, Code, Tools | 128,000 tokens | Flagship frontier model for complex reasoning |
| **Mistral Small** | `mistral-small-latest` | Fast Chat, Tools | 128,000 tokens | Low-latency, cost-effective inference |
| **Codestral** | `codestral-latest` | Code Generation, FIM | 256,000 tokens | Specialized programming and code completion |
| **Pixtral 12B** | `pixtral-12b-2409` | Vision, Multimodal | 128,000 tokens | Multimodal document and image understanding |
| **Mistral Embed**| `mistral-embed` | Embeddings | 8,192 tokens | High-performance 1024-dimensional dense embeddings |
---
## Capabilities & Protocol Translation
### Role Normalization
Mistral's chat completion endpoints require standard `system`, `user`, `assistant`, and `tool` roles. When clients send requests with OpenAI's newer `developer` role, GoRoute automatically normalizes it to `system` before dispatching to Mistral.
### Model Auto-Discovery
The Mistral adapter implements `ListModels` (`internal/adapter/mistral/adapter.go:ListModels`), querying `GET /v1/models` using your configured credential. When model discovery is enabled in GoRoute (`discovery.enabled: true`), new models released by Mistral are automatically cataloged into the gateway's registry.
### Function & Tool Calling
Mistral supports native tool calling identical to OpenAI function calling specifications. Tool calls, tool results, and `tool_choice` parameters are forwarded transparently.
---
## Gateway Configuration
Configure Mistral deployments in `configs/config.yaml`:
```yaml
model_list:
# Primary: Mistral Large
- model_name: mistral-large
routing_strategy: weighted-shuffle
fallbacks: ["mistral-small"]
model_params:
model: mistral/mistral-large-latest
api_key: os.environ/MISTRAL_API_KEY
rpm: 1000
tpm: 500000
# Dedicated Code Assistant: Codestral
- model_name: codestral
model_params:
model: mistral/codestral-latest
api_key: os.environ/MISTRAL_API_KEY
# Multimodal: Pixtral 12B
- model_name: pixtral
model_params:
model: mistral/pixtral-12b-2409
api_key: os.environ/MISTRAL_API_KEY
# Dense Vector Embeddings
- model_name: mistral-embed
model_params:
model: mistral/mistral-embed
api_key: os.environ/MISTRAL_API_KEY
```
Set the API key in your environment:
```bash
export MISTRAL_API_KEY="your-mistral-api-key"
```
---
## Usage Examples
### 1. Chat Completion via cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "mistral-large",
"messages": [
{"role": "system", "content": "You are a senior systems engineer."},
{"role": "user", "content": "Compare eBPF and kernel modules for network observability."}
],
"temperature": 0.2,
"max_tokens": 500
}'
```
### 2. Code Generation with Codestral
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "codestral",
"messages": [
{"role": "user", "content": "Write an efficient LRU Cache in Go using sync.Mutex and container/list."}
],
"temperature": 0.1
}'
```
### 3. Embeddings via Mistral Embed
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "mistral-embed",
"input": ["Mistral Embed provides 1024-dimensional semantic representations."]
}'
```
### 4. Calling Mistral via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "mistral-chat",
Model: "mistral/mistral-large-latest",
APIKey: "os.environ/MISTRAL_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Failed to initialize gateway: %v", err)
}
defer gw.Close()
response, err := gw.Chat(ctx, "mistral-chat", "What makes European data privacy regulations unique?")
if err != nil {
log.Fatalf("Mistral chat error: %v", err)
}
fmt.Println("Mistral Large Response:\n", response)
}
```
> [!NOTE]
> Mistral endpoints deployed in your own VPC or hosted on Azure AI Model Catalog can be targeted by specifying a custom `api_base` within `model_params`.
---
## Ollama
URL: https://goroute-ai.github.io/docs/providers/ollama
Markdown: https://goroute-ai.github.io/docs/providers/ollama.md
Section: Control Plane Documentation > Choose a model provider
# Ollama
GoRoute provides a native adapter for **Ollama** (`internal/adapter/ollama/`), allowing development teams and edge deployments to route requests to local or self-hosted open-source models with zero egress cost and complete data privacy.
The adapter connects to Ollama's native HTTP API (`/api/chat`, `/api/embed`, `/api/tags`), automatically converting between OpenAI's request/response formats and Ollama's schema while handling newline-delimited JSON (NDJSON) streaming translation.
---
## Architecture & Stream Translation
Ollama streams response tokens as newline-delimited JSON (NDJSON) rather than the standard SSE (`text/event-stream`) format expected by OpenAI clients.
GoRoute implements native stream transcoding in `internal/adapter/ollama/adapter.go:StreamChatCompletion` using `sse.ReadNDJSON`:
```mermaid
flowchart LR
Client["Client / Application"] -->|"POST /v1/chat/completions (SSE)"| GW["GoRoute AI Gateway"]
GW -->|"POST /api/chat (stream)"| Ollama["Ollama Daemon (:11434)"]
Ollama -->|"NDJSON Lines"| GW
GW -->|"Standard OpenAI SSE"| Client
```
When Ollama indicates stream completion (`done: true`), GoRoute captures `prompt_eval_count` and `eval_count` and attaches a final standard `usage` object to the SSE stream before emitting `data: [DONE]`.
---
## Local Model Auto-Discovery
The Ollama adapter implements `ListModels` by querying Ollama's local catalog endpoint:
```text
GET /api/tags
```
When background model synchronization is enabled in GoRoute (`discovery.enabled: true`), any model pulled to your local daemon (e.g., `ollama run deepseek-r1:14b` or `ollama pull llama3.2:3b`) is automatically discovered and registered with the gateway.
---
## Network Security & SSRF Configuration
Because Ollama typically runs over plaintext HTTP on `http://localhost:11434` or internal private IP addresses, GoRoute's SSRF protection policies must be configured accordingly in `configs/config.yaml`:
> [!IMPORTANT]
> In production environments, GoRoute blocks private IP ranges and unencrypted HTTP by default. To connect to a local or internal Ollama server, explicitly permit private networks and HTTP in the security configuration:
```yaml
security:
network:
outbound_ssrf:
allow_private_networks: true
allow_insecure_http: true # Required for http:// endpoints
allowed_schemes: ["http", "https"]
allowed_private_cidrs:
- "127.0.0.1/32"
- "10.0.0.0/8"
- "172.16.0.0/12"
```
If GoRoute runs inside Docker or Kubernetes:
- `http://localhost:11434` points to the container itself.
- Use `http://host.docker.internal:11434` (macOS/Windows Docker) or your cluster DNS service (e.g. `http://ollama.ollama.svc.cluster.local:11434`).
---
## Gateway Configuration
Configure local Ollama models in `configs/config.yaml`:
```yaml
model_list:
# Local Llama 3.2 3B for offline/edge development
- model_name: local-llama
model_params:
model: ollama/llama3.2:3b
api_base: "http://localhost:11434"
# Local DeepSeek R1 14B for offline reasoning
- model_name: local-deepseek
model_params:
model: ollama/deepseek-r1:14b
api_base: "http://localhost:11434"
# Local Nomic Embeddings
- model_name: local-embed
model_params:
model: ollama/nomic-embed-text
api_base: "http://localhost:11434"
```
---
## Usage Examples
### 1. Chat Completion via cURL
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "local-llama",
"messages": [
{"role": "user", "content": "Explain the difference between TCP and UDP in one sentence."}
],
"temperature": 0.2
}'
```
### 2. Streaming Chat with Token Generation
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "local-llama",
"messages": [{"role": "user", "content": "Write a quicksort implementation in Python."}],
"stream": true
}'
```
### 3. Local Embeddings via Ollama
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "local-embed",
"input": ["Self-hosted vector embeddings for private search."]
}'
```
### 4. Calling Ollama via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "local-llama",
Model: "ollama/llama3.2:3b",
APIBase: "http://localhost:11434",
},
),
)
if err != nil {
log.Fatalf("Gateway initialization failed: %v", err)
}
defer gw.Close()
reply, err := gw.Chat(ctx, "local-llama", "Why are local LLMs important for data security?")
if err != nil {
log.Fatalf("Ollama chat failed: %v", err)
}
fmt.Println("Local Model Output:\n", reply)
}
```
> [!TIP]
> Ollama models can participate directly in high-availability fallback chains. You can configure cloud models (such as GPT-4o or Claude 3.5 Sonnet) to fail over automatically to an on-premises Ollama instance when WAN connectivity is severed.
---
## OpenAI-Compatible Dialects
URL: https://goroute-ai.github.io/docs/providers/openai-compatible
Markdown: https://goroute-ai.github.io/docs/providers/openai-compatible.md
Section: Control Plane Documentation > Choose a model provider
# OpenAI-Compatible Dialects
The modern AI ecosystem standardizes heavily on the OpenAI REST protocol (`POST /v1/chat/completions`, `POST /v1/embeddings`, `GET /v1/models`). However, individual vendors exhibit subtle dialect variations: differences in parameter naming (`max_completion_tokens` vs `max_tokens`), reasoning stream extraction (`delta.reasoning_content`), strict temperature limits, and custom HTTP authentication headers.
GoRoute includes a dedicated, certified **OpenAI-Compatible Dialect Engine** (`internal/adapter/openaicompat/`) containing **60+ pre-certified built-in and extended dialect profiles** (`builtin_profiles.go` and `extended_profiles.go`) as part of the **73 `ProviderType` profiles** registered in `AllProviders()` (`internal/domain/inference.go`), plus `CustomOpenAICompatible` for custom endpoints.
---
## Dialect Engine Architecture
The dialect engine is built around the `DialectProfile` and `ProfileRegistry` structs (`internal/adapter/openaicompat/profile.go` and `registry.go`):
```mermaid
flowchart TD
Client["Client / SDK"] -->|Standard OpenAI Chat Request| Router["GoRoute Routing Engine"]
Router --> Dialect["Dialect Profile Engine\n(internal/adapter/openaicompat)"]
subgraph DialectEngine["Dialect Profile Pipeline"]
direction TB
ParamMap["Parameter Translation\n(max_completion_tokens → max_tokens)"]
ConstraintCheck["Constraint Enforcement\n(Temperature range, TopP rejection)"]
AuthInjection["Authentication Strategy\n(Bearer / Header)"]
StreamAdapter["Streaming Adapter\n(ExtractReasoningDelta)"]
end
Dialect --> DialectEngine
DialectEngine --> Upstream["Certified Upstream Provider\n(Groq, Cerebras, Together, vLLM, etc.)"]
```
### Core Dialect Capabilities
1. **Parameter Translation**: Maps non-standard parameters dynamically (e.g. converting `max_completion_tokens` to `max_tokens` for providers that reject the newer OpenAI property).
2. **Reasoning Content Extraction (`ExtractReasoningDelta`)**: Intercepts streaming chunks containing extended thinking tokens (e.g., DeepSeek R1, Grok, SambaNova) and exposes them uniformly via `delta.reasoning_content`.
3. **Dialect Constraints**: Automatically clamps or removes invalid parameters based on model family (e.g., stripping `temperature` and `top_p` for reasoning models like DeepSeek-R1 or Groq distill, or stripping `stream_options` when `SupportsStreamOptions` is `false`).
4. **Operation Enforcement (`SupportedOperations`)**: Profiles can declare explicit `ProviderOperations` (`Chat`, `Completion`, `Responses`, `Embeddings`, `Rerank`, etc.) enforced by `OperationSupportChecker` before dispatch.
5. **Zero-Latency Model Discovery**: Handles `GET /models` probes dynamically when `SupportsListModels: true`.
---
## Certified Provider Catalog
The 73 `ProviderType` profiles supported across GoRoute (spanning native adapters and 60+ built-in/extended OpenAI-compatible profiles in `builtin_profiles.go` and `extended_profiles.go`, plus `CustomOpenAICompatible`) include the following core dialect categories:
### 1. Ultra-Fast Wafer-Scale & Hardware Acceleration
Engineered for extreme token generation speeds (hundreds to thousands of tokens per second):
| Provider | ID | Default Base URL | Reasoning Delta | Tools | Key Models |
| :--- | :--- | :--- | :---: | :---: | :--- |
| **Cerebras** | `cerebras` | `https://api.cerebras.ai/v1` | No | Yes | `llama3.3-70b`, `llama3.1-8b` |
| **Groq** | `groq` | `https://api.groq.com/openai/v1` | Distill | Yes | `llama-3.3-70b-versatile`, `mixtral-8x7b-32768` |
| **SambaNova** | `sambanova` | `https://api.sambanova.ai/v1` | Yes | Yes | `Meta-Llama-3.3-70B-Instruct`, `DeepSeek-R1` |
### 2. High-Performance Serverless Inference
Low-latency, cost-effective serverless inference providers:
| Provider | ID | Default Base URL | Reasoning Delta | Embeddings | Key Models |
| :--- | :--- | :--- | :---: | :---: | :--- |
| **Fireworks AI** | `fireworks` | `https://api.fireworks.ai/inference/v1` | No | Yes | `llama-v3p3-70b-instruct`, `qwen2p5-coder-32b-instruct` |
| **Together AI** | `together` | `https://api.together.xyz/v1` | No | Yes | `meta-llama/Llama-3.3-70B-Instruct-Turbo` |
| **DeepInfra** | `deepinfra` | `https://api.deepinfra.com/v1/openai` | No | Yes | `meta-llama/Meta-Llama-3.1-70B-Instruct` |
| **Hyperbolic** | `hyperbolic` | `https://api.hyperbolic.xyz/v1` | Yes | No | `deepseek-ai/DeepSeek-V3`, `DeepSeek-R1` |
| **Novita AI** | `novita` | `https://api.novita.ai/v3/openai` | Yes | Yes | `deepseek/deepseek-r1`, `llama-3.3-70b` |
| **Friendli AI** | `friendli` | `https://inference.friendli.ai/v1` | No | No | `meta-llama-3.1-70b-instruct` |
| **Nebius AI Studio** | `nebius` | `https://api.studio.nebius.ai/v1` | Yes | No | `deepseek-ai/DeepSeek-R1`, `meta-llama/Meta-Llama-3.1-70B` |
| **Scaleway** | `scaleway` | `https://api.scaleway.ai/v1` | No | No | `llama-3.3-70b-instruct` |
| **Lambda Labs** | `lambda` | `https://api.lambdalabs.com/v1` | No | No | `hermes-3-llama-3.1-405b` |
### 3. Search & Reasoning
Engines featuring integrated web search or deep multi-step reasoning capabilities:
| Provider | ID | Default Base URL | Reasoning Delta | Special Capabilities |
| :--- | :--- | :--- | :---: | :--- |
| **xAI (Grok)** | `xai` | `https://api.x.ai/v1` | Yes | Grok 2, Grok 2 Vision, reasoning output |
| **Perplexity AI**| `perplexity` | `https://api.perplexity.ai` | No | Live citation generation (`sonar-pro`, `sonar`) |
| **SiliconFlow** | `siliconflow` | `https://api.siliconflow.cn/v1` | Yes | DeepSeek R1/V3, Qwen 2.5, BGE reranker & embeddings |
| **DeepSeek** | `deepseek` | `https://api.deepseek.com/v1` | Yes | DeepSeek-V3, DeepSeek-R1 reasoning models |
### 4. Multi-Model Aggregators
Dynamic routing across hundreds of upstream endpoints:
| Provider | ID | Default Base URL | Reasoning Delta | Description |
| :--- | :--- | :--- | :---: | :--- |
| **OpenRouter** | `openrouter` | `https://openrouter.ai/api/v1` | Yes | Universal aggregator with auto-failover and model routing |
| **Anyscale** | `anyscale` | `https://api.endpoints.anyscale.com/v1` | No | Scalable foundation model hosting powered by Ray |
### 5. Self-Hosted & Local Inference Engines
Run models on internal GPU clusters or local developer workstations:
| Provider | ID | Default Base URL | Supported Frameworks |
| :--- | :--- | :--- | :--- |
| **vLLM** | `vllm` | `http://localhost:8000/v1` | PagedAttention, Tensor Parallelism, Guided Decoding |
| **SGLang** | `vllm` (compatible) | Custom URL | RadixAttention for fast prefix caching |
| **TensorRT-LLM** | `vllm` (compatible) | Custom URL | NVIDIA optimized execution engine |
| **Baseten** | `baseten` | `https://bridge.baseten.co/v1` | Truss-packaged custom model deployments |
| **LMStudio / Jan / LocalAI** | `ollama` or `custom` | Local port | Desktop developer sandboxes |
### 6. Specialized Embeddings & Reranking
Optimized exclusively for dense retrieval and ranking:
| Provider | ID | Default Base URL | Modality | Top Models |
| :--- | :--- | :--- | :--- | :--- |
| **Voyage AI** | `voyage` | `https://api.voyageai.com/v1` | Embeddings | `voyage-3`, `voyage-3-large`, `voyage-code-2` |
| **Jina AI** | `jina` | `https://api.jina.ai/v1` | Embeddings & Rerank | `jina-embeddings-v3`, `jina-reranker-v2-base-multilingual` |
### 7. Enterprise Foundation Model Platforms
Enterprise AI platforms with custom SLAs and hardware governance:
| Provider | ID | Default Base URL | Modality | Top Models |
| :--- | :--- | :--- | :--- | :--- |
| **NVIDIA NIM** | `nvidia` | `https://integrate.api.nvidia.com/v1` | Chat, Vision | `meta/llama-3.3-70b-instruct`, `mistralai/mistral-large-2` |
| **Replicate** | `replicate` | `https://api.replicate.com/v1` | Chat, Multi | Hosted open-source model deployments |
| **AI21 Labs** | `ai21` | `https://api.ai21.com/studio/v1` | Chat | `jamba-1.5-large`, `jamba-1.5-mini` |
### 8. Global Enterprise & Regional Providers
Tier-1 cloud platforms across Asia, Europe, and North America:
| Provider | ID | Default Base URL | Features |
| :--- | :--- | :--- | :--- |
| **Cloudflare Workers AI** | `cloudflare` | `https://api.cloudflare.com/client/v4/ai/v1` | Serverless edge inference across 300+ PoPs |
| **Databricks Model Serving** | `databricks` | Configured per workspace | Unity Catalog governance and Mosaic AI |
| **Snowflake Cortex** | `snowflake` | Configured per account | In-database LLM inference and search |
| **Alibaba DashScope** | `dashscope` | `https://dashscope.aliyuncs.com/compatible-mode/v1` | Qwen 2.5 series (`qwen-max`, `qwen-plus`) |
| **ByteDance Volcengine** | `volcengine` | `https://ark.cn-beijing.volces.com/api/v3` | Doubao models, temperature clamped `[0.0, 1.0]` |
| **Zhipu AI BigModel** | `bigmodel` | `https://open.bigmodel.cn/api/paas/v4` | GLM-4 series with web search |
| **Moonshot AI (Kimi)** | `moonshot` | `https://api.moonshot.cn/v1` | Long-context Kimi models (up to 2M tokens) |
| **MiniMax** | `minimax` | `https://api.minimax.chat/v1` | MiniMax-Text-01 with reasoning extraction |
---
## Configuration Examples
### Dialect Configuration in `configs/config.yaml`
```yaml
model_list:
# 1. Cerebras Ultra-Fast Wafer-Scale
- model_name: ultra-fast
model_params:
model: cerebras/llama3.3-70b
api_key: os.environ/CEREBRAS_API_KEY
weight: 100
# 2. xAI Grok with Reasoning
- model_name: grok-chat
model_params:
model: xai/grok-2-1212
api_key: os.environ/XAI_API_KEY
# 3. Together AI with Temperature Clamping
- model_name: together-llama
model_params:
model: together/meta-llama/Llama-3.3-70B-Instruct-Turbo
api_key: os.environ/TOGETHER_API_KEY
# 4. DeepSeek R1 with Reasoning Delta Extraction
- model_name: deepseek-reasoner
model_params:
model: deepseek/deepseek-reasoner
api_key: os.environ/DEEPSEEK_API_KEY
# 5. NVIDIA NIM API
- model_name: nim-llama
model_params:
model: nvidia/meta/llama-3.3-70b-instruct
api_key: os.environ/NVIDIA_API_KEY
# 6. Voyage AI Dedicated Embeddings
- model_name: voyage-embed
model_params:
model: voyage/voyage-3
api_key: os.environ/VOYAGE_API_KEY
```
---
## Reasoning Delta Extraction (`ExtractReasoningDelta`)
When calling reasoning-oriented models (such as `DeepSeek-R1`, `Grok-2`, or SambaNova's R1 endpoint) via streaming, the provider returns reasoning thoughts in a non-standard JSON delta field:
```json
{"choices": [{"delta": {"reasoning_content": "Let's first decompose the algorithm into three phases..."}}]}
```
GoRoute's dialect adapter automatically detects `ExtractReasoningDelta: true`, preserves the field, and streams it to the client:
```bash
curl -N -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-reasoner",
"messages": [{"role": "user", "content": "Solve the 8-queens problem with backtracking."}],
"stream": true
}'
```
```text
data: {"id":"chatcmpl-912","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning_content":"Phase 1: Board representation..."}}]}
...
data: {"id":"chatcmpl-912","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Here is the Python implementation:"}}]}
```
---
## Dialect Parameter Clamping & Overrides
Certain providers enforce strict constraints on parameters:
- **Together AI**: Restricts `temperature` strictly between `0.0` and `1.0`. Any request with `temperature: 1.5` is clamped to `1.0` before transmission.
- **DeepSeek R1 / Grok Distill**: Rejects requests that pass custom `temperature` or `top_p`. The dialect adapter strips these parameters cleanly to prevent upstream HTTP 400 errors.
- **Standard Param Map**: Translates `max_completion_tokens` to `max_tokens` for providers that have not adopted OpenAI's August 2024 spec changes.
> [!TIP]
> If you run a custom internal model that exhibits idiosyncratic parameter requirements, configure a profile in `providers.custom_openai_compatible` in `configs/config.yaml`. See the [Custom Provider Guide](https://goroute-ai.github.io/docs/providers/custom-provider) for syntax details.
---
## Custom In-House Providers
URL: https://goroute-ai.github.io/docs/providers/custom-provider
Markdown: https://goroute-ai.github.io/docs/providers/custom-provider.md
Section: Control Plane Documentation > Choose a model provider
# Custom In-House Providers
GoRoute enables enterprises to onboard custom in-house inference clusters, private endpoints, and self-hosted model engines (such as **vLLM**, **SGLang**, **TensorRT-LLM**, **TGI**, and internal gateways) without writing custom Go code or recompiling the binary.
Custom providers are registered declaratively in `configs/config.yaml` through the certified OpenAI-compatible dialect subsystem (`internal/adapter/openaicompat/registry.go:RegisterCustom`).
---
## Declarative Provider Registration (`providers.custom_openai_compatible`)
To register a custom provider, declare it under the `providers.custom_openai_compatible` section of your configuration file:
```yaml
providers:
custom_openai_compatible:
# 1. In-House vLLM GPU Cluster
- id: "internal-vllm"
display_name: "Internal vLLM H100 Cluster"
base_url: "https://vllm.internal.corp.net/v1"
supports_streaming: "true"
supports_tools: "true"
supports_structured_outputs: "true"
supports_list_models: true
default_headers:
X-Gateway-Identity: "goroute-prod"
X-Internal-Tenant: "engineering"
# 2. On-Premises TensorRT-LLM Engine
- id: "tensorrt-cluster"
display_name: "On-Premises TensorRT-LLM"
base_url: "http://10.200.4.15:8000/v1"
supports_streaming: "true"
supports_tools: "false"
supports_structured_outputs: "true"
supports_list_models: false
default_headers:
X-Acceleration-Engine: "TensorRT-LLM"
```
### Configuration Attributes Reference
| Field | Type | Required | Description |
| :--- | :--- | :---: | :--- |
| `id` | `string` | **Yes** | Unique identifier (must match `^[a-z0-9_\-]+$`). Used as the model prefix (e.g. `internal-vllm/llama-3.3-70b`). |
| `display_name` | `string` | No | Human-readable description displayed in the Operator Web UI. |
| `base_url` | `string` | **Yes** | Root HTTP/HTTPS endpoint of the inference server (including `/v1`). |
| `supports_streaming` | `string` / `bool` | No | Whether the upstream supports Server-Sent Events (SSE). Defaults to `true`. |
| `supports_tools` | `string` / `bool` | No | Whether the upstream supports function and tool calling. |
| `supports_structured_outputs` | `string` / `bool` | No | Whether the upstream supports `response_format: {type: "json_object"}`. |
| `supports_list_models` | `bool` | No | Whether GoRoute can query `GET /models` for auto-discovery. |
| `default_headers` | `map[string]string` | No | Custom HTTP headers injected into every request sent to this provider. |
---
## Registration Lifecycle & SSRF Validation
When GoRoute parses custom provider entries during startup, it passes each profile through the `ProfileRegistry.RegisterCustom` validator:
```mermaid
flowchart TD
Config["configs/config.yaml\nproviders.custom_openai_compatible"] --> Parser["Configuration Parser"]
Parser --> IDCheck{"Validate ID\n^[a-z0-9_\\-]+$"}
IDCheck -->|Invalid| Err1["Fatal: Invalid ID"]
IDCheck -->|Valid| URLCheck{"Validate Base URL & Scheme"}
URLCheck -->|Not HTTP/HTTPS| Err2["Fatal: Invalid Scheme"]
URLCheck -->|Valid| SSRF{"SSRF Policy Verification\n(r.ssrfPolicy.ValidateURL)"}
SSRF -->|Private IP / Prohibited CIDR| Err3["Fatal: Rejected by Outbound SSRF Policy"]
SSRF -->|Passed| Registry["Register into Dialect Profile Engine\n(New DialectProfile + Standard Strategies)"]
Registry --> Ready["Ready for Model List & Invocations"]
```
### SSRF & Network Hardening for Internal Endpoints
Because custom in-house clusters often reside on private corporate subnets (e.g., `10.0.0.0/8`, `172.16.0.0/12`), you must update GoRoute's outbound network policy:
```yaml
security:
network:
outbound_ssrf:
allow_private_networks: true
allow_insecure_http: true # Required if internal cluster uses http:// instead of https://
allowed_schemes: ["http", "https"]
allowed_private_cidrs:
- "10.0.0.0/8"
- "172.16.0.0/12"
blocked_ips:
- "169.254.169.254" # Always block AWS/GCP cloud metadata IP
- "fd00:ec2::254"
```
> [!CAUTION]
> Never disable cloud metadata blocking (`169.254.169.254`). GoRoute protects against SSRF vulnerabilities that could leak host credentials or cloud IAM tokens to unauthorized callers.
---
## Configuring Models for Custom Providers
Once registered, models from your custom provider can be referenced directly in `model_list`:
```yaml
model_list:
# Using the registered custom provider ID as a prefix
- model_name: inhouse-llama
routing_strategy: weighted-shuffle
fallbacks: ["cloud-fallback"]
model_params:
model: internal-vllm/meta-llama/Llama-3.3-70B-Instruct
api_key: os.environ/INTERNAL_VLLM_API_KEY # Optional if cluster requires auth
weight: 100
# Direct ad-hoc override via api_base (without registering a custom provider)
- model_name: adhoc-sglang
model_params:
model: openai/meta-llama/Llama-3.1-8B-Instruct
api_base: "http://sglang-service.internal:30000/v1"
api_key: "dummy-key"
# Cloud backup fallback
- model_name: cloud-fallback
model_params:
model: groq/llama-3.3-70b-versatile
api_key: os.environ/GROQ_API_KEY
```
---
## Usage Examples
### 1. Invocations via cURL
Send standard OpenAI requests referencing your custom model alias:
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "inhouse-llama",
"messages": [
{"role": "system", "content": "You are a private corporate coding assistant."},
{"role": "user", "content": "Explain our internal microservice authentication flow."}
],
"temperature": 0.2
}'
```
### 2. Direct Dynamic Prefix Invocations
If callers have permissions to direct-route, they can prefix the registered provider ID dynamically:
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "internal-vllm/meta-llama/Llama-3.3-70B-Instruct",
"messages": [{"role": "user", "content": "Perform sanity check."}]
}'
```
### 3. Calling Custom In-House Providers via the Go SDK
```go
package main
import (
"context"
"fmt"
"log"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "inhouse-llama",
Model: "openai/meta-llama/Llama-3.3-70B-Instruct",
APIBase: "https://vllm.internal.corp.net/v1",
APIKey: "os.environ/INTERNAL_VLLM_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Failed to initialize gateway: %v", err)
}
defer gw.Close()
response, err := gw.Chat(ctx, "inhouse-llama", "Draft an internal architectural decision record (ADR).")
if err != nil {
log.Fatalf("Custom model chat error: %v", err)
}
fmt.Println("Custom Model Response:\n", response)
}
```
> [!TIP]
> If your internal model engine serves streaming responses, GoRoute translates the stream into standard SSE without buffering, maintaining real-time Time-to-First-Token (TTFT) metrics for internal applications.
---
## AI Governance: Security, Policies & Compliance
URL: https://goroute-ai.github.io/docs/governance
Markdown: https://goroute-ai.github.io/docs/governance.md
Section: Control Plane Documentation > Protect requests and control access
# AI Governance: Security, Policies & Compliance
GoRoute governance gives teams control over who can use AI, which models they can reach, and how requests are checked. It also protects provider credentials and limits where the gateway can send traffic. The guides below explain the safeguards and show how to configure them.
```mermaid
flowchart LR
request["Incoming request"] --> access{"Access allowed?"}
access -->|Yes| policy["Apply request policies"] --> provider["Approved provider"]
access -->|No| deny["Reject request"]
classDef appStyle fill:#153451,stroke:#60A5FA,color:#EFF6FF
classDef gatewayStyle fill:#0B5668,stroke:#22D3EE,color:#ECFEFF
classDef policyStyle fill:#1D5147,stroke:#5EEAD4,color:#ECFDF5
classDef serviceStyle fill:#382F6B,stroke:#A78BFA,color:#F5F3FF
class request appStyle
class access gatewayStyle
class policy policyStyle
class provider serviceStyle
class deny gatewayStyle
```
---
## Core Capabilities
| Capability | Implementation | Guide |
| :--- | :--- | :--- |
| **4-Layer Policy Model** | Hierarchical fail-closed scoping across **Organization → Team → User → Virtual Key** for model allowlists, budgets, and rate limits. | [RBAC & Credential Vault](https://goroute-ai.github.io/docs/governance/rbac-and-vault) |
| **Operator RBAC & OIDC/JWT** | Role-based access control for `/admin/v1/*` and `/ui/*` supporting Master Admin Keys as well as OIDC/JWT identity mapping. | [RBAC & Credential Vault](https://goroute-ai.github.io/docs/governance/rbac-and-vault) |
| **AES-256-GCM Credentials Vault** | Upstream provider API keys, MCP credentials, and guardrail secrets are encrypted at rest and in transit via `internal/storage/vault.go` with DEK rotation. | [RBAC & Credential Vault](https://goroute-ai.github.io/docs/governance/rbac-and-vault) |
| **Virtual Keys (`sk-gw-...`)** | Scoped, revocable data-plane tokens accepted via `Authorization: Bearer` or `x-api-key` (with conflict rejection on mismatched dual headers). | [Virtual Keys](https://goroute-ai.github.io/docs/governance/virtual-keys) |
| **Two-Layer Guardrails Engine (19 Drivers)** | Pipeline v2 with 7 hooks (`pre_call`, `during_call`, `post_call`, `logging_only`, `pre_mcp_call`, `during_mcp_call`, `post_mcp_call`), `POST /v1/guardrails/apply`, Starlark `custom_code`, `WithGuardrailScanner`, streaming windowing, and hierarchical bindings (`Org` → `Team` → `VirtualKey` → `EndUser` → `ModelGroup` → `Tag`). | [Guardrails Engine](https://goroute-ai.github.io/docs/governance/guardrails) |
| **Outbound SSRF Firewall** | `internal/security/network` `SafeTransport` enforces DNS pinning, private-IP/loopback blocking, and link-local address protection on all provider, OTLP, and MCP egress. | [SSRF & Network Security](https://goroute-ai.github.io/docs/governance/network-and-ssrf) |
| **Security Headers & CORS** | Strict HTTP security headers, configurable CORS origins, and production secret validation before startup. | [Production Readiness Checklist](https://goroute-ai.github.io/docs/deploy/production-checklist) |
---
## Fail-Closed Enforcement Flow
```mermaid
flowchart TD
Req["Incoming Request"] --> Auth["Virtual Key (sk-gw-...) or OIDC/JWT"]
Auth --> RBAC["4-Layer RBAC Check
(Org → Team → User → Key)"]
RBAC --> GuardIn["Input Guardrails
(enforce or logging_only)"]
GuardIn --> Vault["Decrypt Upstream Key
(AES-256-GCM Vault)"]
Vault --> SSRF["Outbound SSRF Firewall
(DNS Pinning & CIDR Blocklist)"]
SSRF --> GuardOut["Streaming Output Guardrails
(Chunk Filter & Redaction)"]
```
---
## Explore Governance Guides
- [Virtual Keys](https://goroute-ai.github.io/docs/governance/virtual-keys)
- [RBAC & Credential Vault](https://goroute-ai.github.io/docs/governance/rbac-and-vault)
- [Guardrails Engine](https://goroute-ai.github.io/docs/governance/guardrails)
- [SSRF & Network Security](https://goroute-ai.github.io/docs/governance/network-and-ssrf)
- [Production Readiness Checklist](https://goroute-ai.github.io/docs/deploy/production-checklist)
---
## Scoped Virtual Keys & Quotas
URL: https://goroute-ai.github.io/docs/governance/virtual-keys
Markdown: https://goroute-ai.github.io/docs/governance/virtual-keys.md
Section: Control Plane Documentation > Protect requests and control access
# Scoped Virtual Keys & Quotas
GoRoute AI Gateway allows you to issue scoped **Virtual API Keys (`sk-gw-...`)** to applications, teams, or external clients. Virtual keys act as fine-grained policy boundaries that enforce:
- Organization & Team multi-tenancy
- Role-based permissions (Org Admin, Team Admin, Developer, Service Account)
- Strict model allowlists
- Maximum micro-dollar budget caps
- Atomic sliding-window rate limits (RPM and TPM)
- Instant multi-pod key revocation
---
## 1. Issuing a Scoped Virtual Key
#### cURL / Admin API
```bash
curl -X POST http://localhost:8080/admin/v1/keys \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"name": "mobile-checkout-prod",
"org_id": "org_default",
"team_id": "team_mobile",
"role": "developer",
"max_budget_micros": 50000000,
"allowed_models": ["fast", "gemini-2.5-flash"],
"rpm_limit": 600,
"tpm_limit": 1000000
}'
```
#### Go SDK
```go
vk, err := gw.CreateVirtualKey(ctx, gateway.VirtualKeyRequest{
Name: "mobile-checkout-prod",
OrgID: "org_default",
TeamID: "team_mobile",
Role: "developer",
AllowedModels: []string{"fast", "gemini-2.5-flash"},
MaxBudgetMicros: 50000000, // 50,000,000 µ$ ($50.00 USD)
RPMLimit: 600,
TPMLimit: 1000000,
})
if err != nil {
log.Fatal(err)
}
fmt.Println("Issued Token:", vk.Key) // Only visible upon creation!
```
### Creation Response (HTTP 201 Created)
```json
{
"id": "21fc8b10-e633-477d-a3a1-32aa7f325507",
"key": "sk-gw-45ddf8c87e91a2b3c4d5e6f708192a3b",
"key_prefix": "sk-gw-45ddf8c8",
"name": "mobile-checkout-prod",
"role": "developer",
"org_id": "org_default",
"team_id": "team_mobile",
"max_budget_micros": 50000000,
"rpm_limit": 600,
"tpm_limit": 1000000,
"allowed_models": ["fast", "gemini-2.5-flash"],
"is_active": true
}
```
> **Security Note**: The raw token (`sk-gw-...`) is returned **only once** upon generation. GoRoute computes a SHA-256 hash and stores only the digest in the database.
---
## 2. Instant Key Revocation (`POST /admin/v1/keys/{id}/revoke`)
To immediately terminate access for a compromised or deprecated key:
```bash
curl -X POST http://localhost:8080/admin/v1/keys/21fc8b10-e633-477d-a3a1-32aa7f325507/revoke \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me"
```
The gateway instantly evicts the key from in-memory L1 cache and Redis L2 cache across all cluster nodes. Any subsequent requests using that key are rejected with HTTP 401 Unauthorized within microseconds.
---
## RBAC & Key Vault Architecture
URL: https://goroute-ai.github.io/docs/governance/rbac-and-vault
Markdown: https://goroute-ai.github.io/docs/governance/rbac-and-vault.md
Section: Control Plane Documentation > Protect requests and control access
# RBAC & Key Vault Architecture
GoRoute AI Gateway implements a dual-boundary security architecture ensuring cryptographic separation between client virtual keys and upstream provider credentials.
```mermaid
graph TD
Client["Client Token
(Virtual Key sk-gw-...)"] --> Gateway["GoRoute AI Gateway
(Dual-Boundary Isolation)"]
Gateway --> RBAC["4-Layer RBAC Gatekeeper
(Org → Team → Role → Key)"]
RBAC --> Vault["AES-256-GCM Key Vault
(AEAD Envelope Cryptography)"]
Vault --> Decrypted["Decrypted Upstream Secret
(Only in Ephemeral RAM)"]
Decrypted --> Provider["AI Provider API
(OpenAI, Claude, Gemini)"]
```
---
## 1. 4-Layer RBAC Hierarchy
1. **Organization Layer (`OrgID`)**: Enforces strict multi-tenant boundary. Telemetry, logs, and spend queries are strictly isolated to the organization.
2. **Team Layer (`TeamID`)**: Sub-divides organizations into business units (e.g. `checkout`, `fraud`, `analytics`) with allocated monthly budgets.
3. **Role-Based Permissions (`Role`)**:
- `org_admin`: Manage organization settings, issue admin keys, query cross-team spend.
- `team_admin`: Manage team keys, view team spend, adjust team rate limits.
- `developer`: Execute chat completions, stream tokens, call permitted MCP tools.
- `service_account`: High-throughput machine-to-machine integrations.
4. **Key-Level Allowlists (`AllowedModels`) & Fail-Closed Team Allowlists**: Restricts individual keys and teams to specific models, MCP servers, A2A agents, and client apps (`allowed_models`, `allowed_mcp_servers`, `allowed_a2a_agents`, `allowed_client_apps` persisted in PostgreSQL). Team lookup errors fail closed rather than falling back to permissive defaults.
---
## 1a. Control-Plane Operator RBAC & Inbound JWT/OIDC
Control-plane endpoints (`/admin/v1/*`) enforce separate operator authorization from data-plane virtual keys:
- **Operator Roles**: `platform_admin` (full cross-organization administration, required for master admin key), `org_admin` (scoped to a single organization), and `org_viewer` (read-only access within a single organization).
- **Inbound JWT/OIDC**: Operators can authenticate with signed HMAC/OIDC JWTs alongside the Master Admin Key (`platform_admin`).
- **Data-Plane Isolation**: The Master Admin Key cannot be used as a fallback credential on data-plane inference routes (`/v1/*`).
- **Production Secret Validation**: When `server.environment`, `ENV`, or `GATEWAY_ENV` is set to `production` (or `prod`), startup fails immediately if `auth.master_admin_key`, `auth.master_encryption_key`, or `auth.jwt_secret` retains a default value.
- **HTTP Security Headers & CORS**: `SecurityHeadersMiddleware` sets `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`, `Content-Security-Policy`, and opt-in `Strict-Transport-Security` (`server.enable_hsts`), while `CORSMiddleware` disables cross-origin headers unless `server.cors_allowed_origins` is explicitly configured.
---
## 2. AES-256-GCM Envelope Encryption
All upstream provider credentials (e.g. OpenAI API keys, Anthropic secrets) stored in PostgreSQL are encrypted at rest using **AES-256-GCM AEAD (Authenticated Encryption with Associated Data)**:
- Keys are encrypted using a 256-bit Master Encryption Key (`master_encryption_key`).
- A cryptographically random 12-byte initialization vector (nonce) is generated for each record.
- Authentication tags prevent tampering or ciphertext modification.
- Keys are decrypted exclusively in ephemeral RAM during the outbound HTTP request dispatch and immediately zeroed from memory buffers.
---
## 3. Secret Masking in Telemetry & Logs
GoRoute automatically scrubs API keys, bearer tokens, and credentials from all log outputs, error messages, and OpenTelemetry trace attributes:
- `Authorization: Bearer sk-gw-45dd...` -> Masked to `sk-gw-45dd****`
- Upstream provider keys never appear in log statements or JSON payloads.
---
## Organization Guardrails
URL: https://goroute-ai.github.io/docs/governance/guardrails
Markdown: https://goroute-ai.github.io/docs/governance/guardrails.md
Section: Control Plane Documentation > Protect requests and control access
# Organization Guardrails
The gateway supports organization-scoped checks for prompt-injection signatures, blocked keywords, credential-like strings, and selected PII patterns. Policies are managed through the admin API and applied to chat requests. These checks are pattern based; they are not a replacement for application validation, provider safety controls, or a comprehensive DLP system.
## Configure a policy
Use the master admin key to set a policy for an organization:
```bash
curl -X POST http://localhost:8080/admin/v1/organizations/org_default/guardrails \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"mode": "enforce",
"enable_prompt_injection": true,
"prompt_injection_action": "block",
"enable_pii_detection": true,
"pii_action": "redact",
"enable_secret_detection": true,
"secret_action": "redact",
"blocked_keywords": ["restricted-project"],
"redact_output_pii": true
}'
```
Read the saved policy with `GET /admin/v1/organizations/{orgID}/guardrails`.
## Actions, Modes, and Coverage
- **Policy Modes**: Set `"mode": "enforce"` (default) to block or redact violations, or `"mode": "logging_only"` to audit violations in structured logs without blocking or mutating payloads.
- **Actions**: Prompt-injection, secret, and PII checks support `block`, `redact`, or `audit` actions. Blocked keywords block matching content across all scanned roles.
- **Input & Output Coverage**: The engine scans **all message roles** (`system`, `developer`, `user`, `assistant`, `tool`) on supported inference endpoints, blocks or redacts non-streaming outputs, inspects and redacts **streaming SSE chunks** (`internal/guardrails/stream_filter.go`), and enforces inbound/outbound guardrails on **A2A** and **MCP** traffic.
- **Fail-Closed Policy Lookups**: If guardrail policy retrieval fails, the gateway fails closed rather than bypassing safety checks.
- **Provider-Neutral Scanner Hooks**: Embedders using the Go SDK or server can register custom external scanners via `ConfigureGuardrailScanners(timeout, failClosed, scanners...)`.
Guardrail violations are recorded in logs and organization policy changes are written to the audit log. See the [admin API reference](https://goroute-ai.github.io/docs/api/admin-api) and [virtual-key security guide](https://goroute-ai.github.io/docs/governance/virtual-keys).
---
## Two-Layer Guardrails Engine (`v1.8.0`)
In `v1.8.0`, GoRoute expands baseline org policies into a full **Two-Layer Guardrails Engine** supporting tenant-scoped `guardrail_definitions` (AES-256-GCM Vault encrypted) and hierarchical `guardrail_bindings` (`Org` → `Team` → `VirtualKey` → `EndUser` → `ModelGroup` → `Tag`, with `default_on` and anti-bypass `is_locked`):
- **19 Built-In Drivers**: `openai_moderation`, `bedrock` (SigV4), `azure/prompt_shield`, `azure/text_moderations`, `model_armor`, `presidio`, `lakera_v2`, `guardrails_ai`, `pangea`, `local_content_filter`, `generic_http` webhooks, sandboxed Starlark `custom_code` (`50,000`-step cap, zero filesystem/import access, SSRF-checked `http_get`/`http_post`), `tool_permission`, `tool_policy` (cross-turn taint tracking), `block_code_execution`, `mcp_security`, `mcp_end_user_permission`, `llm_as_a_judge`, and `decision_model`.
- **7 Execution Hooks**: `pre_call`, concurrent canceling `during_call`, `post_call`, async `logging_only`, `pre_mcp_call`, `during_mcp_call`, and `post_mcp_call`, plus cross-chunk `StreamWindowEvaluator` masking and streamed `delta.tool_calls` scanning.
- **Standalone Evaluation Endpoint**: `POST /v1/guardrails/apply` evaluates inputs, outputs, or tool calls directly against Pipeline v2 without dispatching to an upstream LLM.
- **Public Go SDK Extensibility**: Register custom in-process scanners via `gateway.WithGuardrailScanner(name, scanner, opts...)` (`allow` / `block` / `flag` / `modify`).
- **Operator Console & Telemetry**: Manage named registries, Starlark rules, tool trust policies, and micro-dollar usage rollups in the 4-Tab Guardrails Console at `/ui/`, backed by `guardrail.evaluate` OpenTelemetry spans and `gateway_guardrail_*` Prometheus metrics.
---
## Network Security & SSRF Protection
URL: https://goroute-ai.github.io/docs/governance/network-and-ssrf
Markdown: https://goroute-ai.github.io/docs/governance/network-and-ssrf.md
Section: Control Plane Documentation > Protect requests and control access
# Network Security & SSRF Protection
When AI agents invoke tools (MCP), fetch web content, or interact with external services, malicious prompts can attempt **Server-Side Request Forgery (SSRF)** to compromise cloud metadata endpoints, internal databases, or private cluster IPs.
GoRoute AI Gateway embeds strict network perimeter defenses to prevent outbound SSRF attacks.
---
## 1. Outbound SSRF Gating
Every outbound HTTP dial executed by GoRoute is inspected by an anti-SSRF dialer:
- **Cloud Metadata Endpoint Blocking**: Outbound calls to `169.254.169.254` (AWS, GCP, Azure, and DigitalOcean instance metadata services) are permanently blocked.
- **Private RFC 1918 CIDR Enforcement**: Outbound requests to internal loopback (`127.0.0.0/8`), private networks (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), and IPv6 site-local addresses (`fc00::/7`, `fe80::/10`) are rejected unless explicitly allowlisted in configuration.
- **DNS Rebinding Prevention**: Validates resolved IP addresses at socket connection time, preventing attackers from using dynamic DNS records to bypass hostname checks.
- **Redirect Limits**: Restricts HTTP redirects to a maximum of 3 hops and enforces that redirect locations do not point to restricted IP ranges.
---
## 2. Inbound Client CIDR Whitelisting
Restrict gateway ingress to trusted network segments or VPN tunnels:
```yaml
security:
network:
outbound_ssrf:
allow_private_networks: false
allowed_schemes: ["https"]
blocked_ips:
- "169.254.169.254"
dial_timeout: 10s
max_redirects: 3
inbound_caller:
expose_on_public_internet: false
allowed_client_cidrs:
- "10.240.0.0/16"
- "192.168.1.0/24"
```
---
## AI FinOps: Cost Management & Optimization
URL: https://goroute-ai.github.io/docs/finops
Markdown: https://goroute-ai.github.io/docs/finops.md
Section: Control Plane Documentation > Manage cost and usage
# AI FinOps: Cost Management & Optimization
When several applications and teams use AI, it can be difficult to see where spend comes from or keep usage within limits. GoRoute records request costs, checks budgets before sending work to a provider, and can enforce request and token limits. The detailed guides explain how this accounting works and what it means for your setup.
---
## Core Capabilities
| Capability | Implementation | Guide |
| :--- | :--- | :--- |
| **64-Bit Integer Micro-Dollar Ledger** | All costs are computed in integer micro-dollars (`1 USD = 1,000,000 micro-dollars`) with intermediate pico-dollar precision and zero `float32`/`float64` accounting. | [Micro-Dollar Ledger](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger) |
| **Cache-Write & Modality Pricing** | Supports prompt-cache write pricing (`CacheCreationTokens` / `cache_write_cost_micros_per_1m`), audio/image/video modality rates (`0007`/`0012`), and immutable `model_pricing_history`. | [Micro-Dollar Ledger](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger) |
| **Unpriced Usage Tracking** | Tracks `unpriced_requests` and `unpriced_tokens` explicitly so custom or uncatalogued models never silently report `$0.00` without visibility. | [Micro-Dollar Ledger](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger) |
| **Pre-Flight Reservations & Lease Renewal** | Reserves upper-bound estimated cost before upstream dispatch, renews leases during long streams, and reconciles exact token spend on completion. | [Micro-Dollar Ledger](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger) |
| **Durable PostgreSQL Outboxes** | Persistent spend (`0008`) and budget-reset (`0011`) outbox tables ensure zero lost accounting events across restarts or transient DB blips. | [Deployment & Storage](https://goroute-ai.github.io/docs/deploy/deployment-and-storage) |
| **Single-Roundtrip Redis Lua Rate Limits** | Atomic sliding-window RPM and TPM enforcement in Redis with fail-closed behavior and `{org:}` hash-tag compatibility for Redis Cluster. | [Sliding-Window Rate Limits](https://goroute-ai.github.io/docs/finops/rate-limiting) |
| **Pre-Flight Tokenizer** | Local zero-network token estimation (`POST /v1/tokenize`, `POST /anthropic/v1/messages/count_tokens`) to enforce TPM/context budgets before dispatch. | [Pre-Flight Tokenizer](https://goroute-ai.github.io/docs/finops/preflight-tokenization) |
---
## FinOps Accounting Lifecycle
```mermaid
sequenceDiagram
participant Client
participant Gateway
participant Redis as Redis (Lua + Leases)
participant PG as PostgreSQL (Outbox + Ledger)
Client->>Gateway: Inference Request (sk-gw-...)
Gateway->>Gateway: Pre-Flight Tokenizer (Estimate Input Tokens)
Gateway->>Redis: Atomic Lua Check (RPM / TPM + Upper-Bound Budget Lease)
Gateway->>Upstream: Dispatch Inference (Renew Lease if Streaming)
Upstream-->>Gateway: Usage Totals (Input, Output, Cache Creation)
Gateway->>Redis: Reconcile Exact Micro-Dollar Spend
Gateway->>PG: Append Spend Event & Immutable Pricing Audit
```
---
## Explore FinOps Guides
- [Micro-Dollar Ledger](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger)
- [Sliding-Window Rate Limits](https://goroute-ai.github.io/docs/finops/rate-limiting)
- [Pre-Flight Tokenizer](https://goroute-ai.github.io/docs/finops/preflight-tokenization)
- [Virtual Keys & Budgets Demo](https://goroute-ai.github.io/docs/examples/virtual-keys-and-budgets)
---
## Cost Accounting and Spend Records
URL: https://goroute-ai.github.io/docs/finops/micro-dollar-ledger
Markdown: https://goroute-ai.github.io/docs/finops/micro-dollar-ledger.md
Section: Control Plane Documentation > Manage cost and usage
# Cost Accounting and Spend Records
The gateway represents calculated cost as integer micro-dollars: `1 USD = 1,000,000 µ$`. The calculator uses integer arithmetic and pricing values from the configured model catalogue. This avoids floating-point rounding in the gateway's accounting calculation; it does not guarantee that provider pricing metadata or reported token usage is complete or current.
## Cost calculation
For a model with pricing metadata, `CalculateCostMicros` (`internal/finops/calculator.go`) accounts for:
- **Non-cached prompt tokens** (`input_cost_micros_per_1m`)
- **Cached prompt read tokens** (`cached_input_cost_micros_per_1m`)
- **Cache-write tokens** (`CacheCreationTokens` billed at `cache_write_cost_micros_per_1m` added in migration `0012`, falling back to `input_cost_micros_per_1m` if unset)
- **Regular completion tokens** (`output_cost_micros_per_1m`)
- **Reasoning tokens** (`reasoning_cost_micros_per_1m`, falling back to `output_cost_micros_per_1m` if unset)
- **Modality-specific units** (migrations `0007` and `0012`): rerank search units (`rerank_cost_micros_per_search_unit`), image generation base/HD (`image_cost_micros_base`, `image_cost_micros_hd`), audio STT per second (`audio_stt_cost_micros_per_second`), audio TTS per 1k characters (`audio_tts_cost_micros_per_1k_chars`), and extensible `modality_pricing_json`, recorded with `modality` and `modality_units` on `spend_logs`.
Each token rate is expressed in micro-dollars per million tokens, and the total numerator is divided by one million with ceiling division so a non-zero priced request does not round down to zero. Upper-bound pre-dispatch budget reservations (`EstimateUpperBoundCostMicros`) also factor in `cache_write_cost_micros_per_1m`.
The gateway returns cost telemetry on inference responses and records spend events for later queries. If a model has no pricing record (or unverified rates), the calculator returns zero cost and increments `unpriced_requests` and `unpriced_tokens` in spend summaries (`/admin/v1/spend/summary`) so unpriced usage is never silently hidden.
## Durable Outbox, Budget Reservations & Period Rollover
- **Pre-Dispatch Upper-Bound Reservations & Lease Renewal**: Before dispatching upstream, the gateway reserves an upper-bound micro-dollar budget in Redis and renews the reservation lease for long-running or streaming requests.
- **Durable Postgres Spend Event Outbox**: Spend events are persisted through a retryable PostgreSQL outbox (`0008_add_spend_event_outbox.up.sql`) and reconciled idempotently in Redis by event ID (`gw:spend:applied:{org}`), preventing double-counting on retries.
- **Unified Modality Budget Gating**: Chat, embeddings, audio, and image requests share unified budget gating with request-sized modality estimates.
- **Automatic Budget Period Rollover**: `ResetExpiredBudgets` advances expired budget windows, synchronizes entity `spent_micros` in PostgreSQL (`0011_budget_reset_outbox.up.sql`), and clears corresponding Redis spend counters.
## Spend persistence
Spend events are queued to a buffered Go channel and processed by a background worker. The worker updates spend counters and batches durable writes to the configured repository. If the queue is full, the worker attempts a synchronous write for that event, so storage backpressure can affect a request in that exceptional case.
Batch size, flush interval, and channel size are configurable. The default values are in the gateway configuration; they are not a throughput guarantee.
## Query spend data
The admin API provides spend summary and spend log endpoints under `/admin/v1/spend`. Authenticate with an admin key:
```bash
curl http://localhost:8080/admin/v1/spend/summary \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
The exact summary fields follow the API response and storage implementation for the gateway version you run. See the [admin API reference](https://goroute-ai.github.io/docs/api/admin-api) and [virtual keys guide](https://goroute-ai.github.io/docs/governance/virtual-keys).
---
## Distributed Rate Limiting (RPM & TPM)
URL: https://goroute-ai.github.io/docs/finops/rate-limiting
Markdown: https://goroute-ai.github.io/docs/finops/rate-limiting.md
Section: Control Plane Documentation > Manage cost and usage
# Distributed Rate Limiting (RPM & TPM)
GoRoute AI Gateway implements an atomic, distributed rate limiter using Redis Lua scripts and a sliding-window counter algorithm. It enforces both **Requests Per Minute (RPM)** and **Tokens Per Minute (TPM)** across multiple gateway pods with microsecond synchronization.
```mermaid
graph TD
Client["Client Request"] --> Limiter["Rate Limiting Middleware"]
Limiter --> Lua["Atomic Redis Lua Script
(Sliding Window Evaluation)"]
Lua --> Check{"Quota Exceeded?"}
Check -- "Yes: RPM/TPM Cap Hit" --> Reject["HTTP 429 Too Many Requests
(Retry-After: 45s)"]
Check -- "No: Quota Available" --> Pass["Pass to Inference Pipeline"]
```
---
## Sliding Window Lua Algorithm
Unlike fixed-window counters that permit 2x traffic bursts across window boundaries, GoRoute's sliding-window algorithm evaluates consumption across a continuously moving 60-second window:
1. Removes expired request timestamps (`ZREMRANGEBYSCORE key 0 (now - window)`).
2. Calculates current usage count within the active window (`ZCARD key`).
3. If current usage exceeds `rpm_limit`, returns remaining cooldown seconds.
4. Otherwise, registers the new request timestamp (`ZADD key now requestId`) and grants access.
---
## Redis Cluster, Hash Tags & Fail-Closed Behavior
- **Redis Cluster & Sentinel (`UniversalClient`)**: `RedisFabric` supports standalone Redis (`storage.redis_addr`), Redis Cluster (`storage.redis_addrs`), and Redis Sentinel (`storage.redis_master_name`). Multi-key Lua scripts use `{org:}` hash tags so all keys for an organization map to the same Redis Cluster slot.
- **Fail-Closed Quota Enforcement**: When distributed Redis quota state is unavailable or exceeds `rate_limit.redis_timeout`, the limiter fails closed rather than allowing unbounded traffic.
---
## HTTP 429 Rate Limit Response
When quota is exceeded, GoRoute returns an informative HTTP 429 response:
```http
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 42
X-RateLimit-Limit-Requests: 100
X-RateLimit-Remaining-Requests: 0
X-RateLimit-Reset-Requests: 42
{
"error": {
"type": "rate_limit_error",
"code": "key_rpm_exceeded",
"message": "Rate limit or quota exceeded (KEY_RPM_EXCEEDED)"
}
}
```
---
## Pre-Flight Tokenizer
URL: https://goroute-ai.github.io/docs/finops/preflight-tokenization
Markdown: https://goroute-ai.github.io/docs/finops/preflight-tokenization.md
Section: Control Plane Documentation > Manage cost and usage
# Pre-Flight Tokenizer
To enforce **Tokens Per Minute (TPM)** quotas *before* dispatching expensive upstream requests, GoRoute AI Gateway runs a fast pre-flight tokenizer.
---
## 1. Why Pre-Flight Tokenization Matters
Without pre-flight token estimation, a client could submit a massive prompt (e.g. 100k tokens) that consumes your entire upstream quota before the gateway can register the spend. By estimating token volume before sending the request upstream, GoRoute reserves tokens in Redis and rejects oversized requests with HTTP 429 before incurring upstream billing.
---
## 2. Token Estimation Heuristics
The pre-flight tokenizer balances speed and accuracy using a zero-allocation scanner:
1. **Exact BPE Encoding**: For OpenAI and Mistral models, uses Byte-Pair Encoding token tables.
2. **Sub-Word Heuristic Fast Path**: For high-throughput scenarios (< 5 µs target overhead), computes token approximations based on character classes, punctuation density, and whitespace boundaries:
```text
Estimated Tokens = (Byte Count / 4) * Complexity Multiplier
```
If the estimated prompt tokens + requested `max_tokens` exceeds the virtual key's available TPM quota, the request is throttled immediately.
---
## AI Agent Management: MCP, Tools & Agent Access
URL: https://goroute-ai.github.io/docs/agents
Markdown: https://goroute-ai.github.io/docs/agents.md
Section: Control Plane Documentation > Connect agents and tools
# AI Agent Management: MCP, Tools & Agent Access
Connect AI agents to tools and other agents through GoRoute. The gateway can bring tools from multiple MCP servers together, limit which callers can use them, and support communication through MCP and A2A. Start with the protocol overview that matches your setup, then follow the related configuration guides.
```mermaid
flowchart LR
agent["AI agent"] --> hub["GoRoute tool hub"] --> tools["Approved tools"]
hub --> peer["Other agents"]
classDef appStyle fill:#153451,stroke:#60A5FA,color:#EFF6FF
classDef gatewayStyle fill:#0B5668,stroke:#22D3EE,color:#ECFEFF
classDef policyStyle fill:#1D5147,stroke:#5EEAD4,color:#ECFDF5
classDef serviceStyle fill:#382F6B,stroke:#A78BFA,color:#F5F3FF
class agent appStyle
class hub gatewayStyle
class tools policyStyle
class peer serviceStyle
```
---
## Core Capabilities
| Capability | Implementation | Guide |
| :--- | :--- | :--- |
| **MCP 2026-07-28 Virtual Hub & REST Bridge** | Aggregates upstream MCP servers into `/mcp`, scoped `/mcp/{serverSlug}`, curated `/mcp/toolsets/{toolsetSlug}`, and REST endpoints (`/mcp-rest/tools/list`, `/mcp-rest/tools/call`). | [MCP Overview](https://goroute-ai.github.io/docs/agents/mcp-overview) · [MCP Hub & REST](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest) |
| **Progressive Tool Discovery** | Serves `/v1/goroute/mcp/registry` so agents load only relevant tool schemas on demand, preventing context-window bloat. | [Progressive Discovery](https://goroute-ai.github.io/docs/agents/mcp-progressive-discovery) |
| **Sandboxed Stdio & SHA-256 Tool Pinning** | Runs local Stdio MCP servers in isolated subprocess sandboxes and pins tool definitions via SHA-256 hashes to prevent rug-pull schema mutations. | [Stdio Sandboxing](https://goroute-ai.github.io/docs/agents/mcp-stdio-sandboxing) |
| **MRTR AEAD Tokens** | Multi-hop tool routing tokens encrypted with AEAD preserve caller identity and permissions across delegated MCP invocations. | [MCP Overview](https://goroute-ai.github.io/docs/agents/mcp-overview) |
| **A2A v1.0 Signed Agent Cards & gRPC** | Serves JCS/JWS-signed Agent Cards (`/.well-known/agent-card.json`), task lifecycle endpoints (`/a2a/v1/agents/{agentSlug}/tasks*`), native gRPC (`server.grpc_port`), and OpenAI bridge (`model: "a2a/{slug}"`). | [A2A Overview](https://goroute-ai.github.io/docs/agents/a2a-overview) · [Agent Cards](https://goroute-ai.github.io/docs/agents/a2a-agent-cards) · [Bridge & gRPC](https://goroute-ai.github.io/docs/agents/a2a-bridge-and-grpc) |
| **Autonomous Multi-Turn Tool Loop** | Executes server-side multi-turn tool-calling loops with bounded iteration budgets and guardrail inspection on every turn. | [Autonomous Tool Loop](https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop) |
| **Coding-Agent & Framework Governance** | Connects Claude Code (`/anthropic`), Cursor, Continue, Cline, Codex CLI, Gemini CLI, OpenWebUI, LangChain, OpenAI Agents SDK, and Google ADK under Virtual Key RBAC. | [Claude Code Integration](https://goroute-ai.github.io/docs/integrations/claude-code) · [Integrations Overview](https://goroute-ai.github.io/docs/integrations/overview) |
---
## Agent & Tool Execution Plane
```mermaid
flowchart TD
Agents["Coding Agents & Frameworks
(Claude Code, Cursor, Codex, ADK, LangChain)"] -->|"Virtual Key (sk-gw-...)"| GW["GoRoute Agent Plane"]
GW -->|"Streamable HTTP / REST"| MCP["MCP Virtual Hub (/mcp, /mcp-rest)"]
MCP --> Stdio["Sandboxed Stdio + SHA-256 Pinning"]
GW -->|"A2A v1.0 / gRPC / model: a2a/{slug}"| A2A["A2A Agent Network
(JCS/JWS Signed Cards)"]
GW -->|"Bounded Iterations"| Loop["Autonomous Tool Loop"]
```
---
## Explore Agent Management Guides
- **MCP**: [MCP Overview](https://goroute-ai.github.io/docs/agents/mcp-overview) · [MCP Hub & REST](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest) · [Progressive Discovery](https://goroute-ai.github.io/docs/agents/mcp-progressive-discovery) · [Stdio Sandboxing](https://goroute-ai.github.io/docs/agents/mcp-stdio-sandboxing)
- **A2A**: [A2A Overview](https://goroute-ai.github.io/docs/agents/a2a-overview) · [Agent Cards](https://goroute-ai.github.io/docs/agents/a2a-agent-cards) · [Bridge & gRPC](https://goroute-ai.github.io/docs/agents/a2a-bridge-and-grpc)
- **Autonomous Execution & Coding Agents**: [Autonomous Tool Loop](https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop) · [Claude Code Integration](https://goroute-ai.github.io/docs/integrations/claude-code) · [Integrations Overview](https://goroute-ai.github.io/docs/integrations/overview)
---
## Model Context Protocol (MCP) Overview
URL: https://goroute-ai.github.io/docs/agents/mcp-overview
Markdown: https://goroute-ai.github.io/docs/agents/mcp-overview.md
Section: Control Plane Documentation > Connect agents and tools
# Model Context Protocol (MCP) Overview
GoRoute AI Gateway is the first enterprise AI data plane with native, multi-version support for the **Model Context Protocol (MCP)** specifications. It unifies distributed tools, databases, APIs, and command-line utilities behind a resilient, high-speed gateway hub.
```mermaid
graph TD
Client["Client / Agent
(Claude Code, Cursor, Custom Agent)"] --> Gateway["GoRoute AI Gateway
(Virtual Hub POST /mcp & REST /mcp-rest)"]
Gateway --> Upstream1["Remote MCP Server A
(Streamable HTTP 2026-07-28 Stateless)"]
Gateway --> Upstream2["Legacy MCP Server B
(HTTP-SSE 2025-11-25 Stateful)"]
Gateway --> Upstream3["Legacy MCP Server C
(HTTP-SSE 2024-11-05 Bidirectional)"]
Gateway --> Upstream4["Local Stdio Tool
(Sandboxed Subprocess)"]
```
---
## 1. Multi-Version Protocol Support
The gateway implements a multi-version transport dispatcher that detects and adapts protocol nuances seamlessly across MCP server generations.
| Specification | Transport Layer | Session Model | Key Characteristics |
| :--- | :--- | :--- | :--- |
| **MCP 2026-07-28** | Streamable HTTP (Stateless) | Stateless | Zero initialization handshake, no session affinity or cookie requirements, direct JSON-RPC 2.0 over HTTP POST. |
| **MCP 2025-11-25** | Streamable HTTP (Stateful) | Stateful Session ID | Requires `Mcp-Session-Id` header tracking, Redis session maps, heartbeat keep-alive, 15-minute inactivity TTL. |
| **MCP 2024-11-05** | HTTP + SSE (Legacy) | Long-Lived Stream | Bidirectional Server-Sent Events stream (`GET /sse`) paired with HTTP POST message submission channel. |
| **Stdio Subprocess** | Standard I/O Pipes | Process-Bound | Executes local command-line tools in isolated child processes with environment scrubbing and binary allowlists. |
---
## 2. Modern 2026-07-28 Streamable HTTP Transport
The modern **2026-07-28 Streamable HTTP** specification represents the cloud-native standard for enterprise tool calling:
- **Zero Protocol-Level Handshake**: Eliminates the mandatory `initialize` $\rightarrow$ `initialized` round-trip handshake prior to calling tools.
- **Direct POST Routing**: Every tool execution is an independent HTTP POST request bearing standard JSON-RPC 2.0 payloads.
- **Horizontal Scalability**: Because requests are completely stateless, any gateway pod in a Kubernetes cluster can handle any tool call without sticky sessions or centralized session affinity routing.
### Example Direct Invocation
```bash
curl -X POST "http://localhost:8080/mcp" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/call",
"params": {
"name": "postgres__run_query",
"arguments": {
"sql": "SELECT COUNT(*) FROM users;"
}
}
}'
```
---
## 3. Legacy 2025 and 2024 Compatibility Adapters
For upstream MCP servers implementing older protocol specifications, the gateway provides automatic backwards-compatibility adapters.
### 2025-11-25 Stateful Adapter (`internal/mcp/adapter_2025.go`)
- **Session Lifecycle Management**: Maps upstream `Mcp-Session-Id` values in Redis with automatic 15-minute inactivity timeouts (`mcp.session_ttl_minutes: 15`).
- **Heartbeat & Keep-Alive Daemon**: Periodically dispatches JSON-RPC ping requests to prevent upstream server timeout disconnects.
- **Principal Digest Validation**: Validates cryptographic hashes of client virtual keys to ensure one client cannot hijack another client's MCP session.
### 2024-11-05 SSE Adapter (`internal/mcp/adapter_2024_sse.go`)
- **Bidirectional SSE Tunneling**: Establishes a persistent `text/event-stream` listener against legacy upstream SSE endpoints, extracting tool emission events into internal Go channels.
- **POST Bridge Routing**: Translates incoming gateway tool calls into upstream SSE postback requests, returning results synchronously to the client.
---
## 4. Virtual Hub Aggregation & Toolsets
Instead of configuring clients with individual URLs for each MCP service, GoRoute AI Gateway acts as an enterprise **Virtual Hub**:
- **Consolidated Endpoint (`POST /mcp`)**: Clients connect to a single gateway endpoint. The gateway queries all registered upstream MCP servers in parallel, merges their tool schemas, and presents a single unified tool manifest.
- **Namespace Collision Prevention**: Avoids naming conflicts across servers by prefixing tool names with the server slug: `{serverSlug}__{toolName}` (e.g. `github__create_issue`, `gitlab__create_issue`).
- **Partial Failure Isolation**: If one upstream MCP server is unreachable or offline, the virtual hub isolates the fault and returns tools from healthy servers without dropping the entire request.
- **Curated Toolsets (`POST /mcp/toolsets/{slug}`)**: Group tools across disparate servers into domain-specific collections (e.g. `devops-tools`, `finance-tools`) restricted by 4-layer RBAC.
---
## 5. Enterprise Reliability & Security Invariants
### Arbitrary Decimal Precision (`big.Rat` & `json.Number`)
Standard JSON parsers convert numeric values into IEEE-754 64-bit floating-point numbers, leading to precision loss in financial transactions, cryptographic IDs, and micro-dollar accounting.
GoRoute AI Gateway parses all MCP tool payloads using Go's `json.Number` and `math/big.Rat`, guaranteeing zero rounding errors or precision drift across high-precision numbers.
### Model-Result-Token-Resumption (MRTR)
For operations returning extensive data sets, the gateway issues single-use, AEAD-encrypted continuation tokens:
- **Identity Bound**: Cryptographically tied to the caller's virtual key and organization ID.
- **TTL Enforced**: Expires automatically after 5 minutes to prevent replay attacks.
- **Memory Optimized**: Large intermediate tool results are spooled to temporary storage rather than held in hot memory.
### Prompt Injection & Tool Pinning Protection
The gateway inspects tool schemas discovered from external servers:
- **Schema Sanitization**: Scans descriptions and parameter names for prompt injection payloads.
- **Cryptographic Tool Pinning**: Computes a SHA-256 fingerprint (`ToolDefHash`) for every registered tool schema. If an upstream server silently mutates its schema without administrative approval, the gateway suspends execution fail-closed.
---
## MCP Virtual Hub & Plain REST Bridge
URL: https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest
Markdown: https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest.md
Section: Control Plane Documentation > Connect agents and tools
# MCP Virtual Hub & Plain REST Bridge
GoRoute AI Gateway provides multiple architectural execution paths for Model Context Protocol (MCP) tools:
1. **Virtual Hub Aggregation (`POST /mcp`)**: Aggregates all registered tools in an organization behind a single JSON-RPC 2.0 endpoint with automatic namespace prefixing.
2. **Scoped Server Execution (`POST /mcp/{slug}`)**: Direct JSON-RPC 2.0 execution routed to a specific registered server without prefix requirements.
3. **Curated Toolsets (`POST /mcp/toolsets/{slug}`)**: Scoped execution across pre-approved, team-specific subsets of tools.
4. **Plain REST Bridge (`/mcp-rest/*`)**: High-performance HTTP REST endpoints (`GET /mcp-rest/tools/list` and `POST /mcp-rest/tools/call`) designed for standard REST clients, completely decoupling applications from JSON-RPC protocol boilerplate.
```mermaid
graph TD
Client["Client / Agent"] --> HubRouter{"Gateway Ingress"}
HubRouter -->|POST /mcp| VHub["Virtual Hub (All Servers Aggregated)"]
HubRouter -->|POST /mcp/{slug}| Scoped["Scoped Server Route"]
HubRouter -->|POST /mcp/toolsets/{slug}| Toolset["Curated Toolset Route"]
HubRouter -->|GET / POST /mcp-rest/*| RESTBridge["Plain REST Bridge"]
VHub --> Dispatcher["Transport Dispatcher"]
Scoped --> Dispatcher
Toolset --> Dispatcher
RESTBridge --> Dispatcher
Dispatcher --> SrvA["MCP Server A"]
Dispatcher --> SrvB["MCP Server B"]
Dispatcher --> SrvC["OpenAPI REST Target"]
```
---
## 1. Virtual Hub Aggregation (`POST /mcp`)
The Virtual Hub merges tools from all active MCP servers in the caller's organization.
### Aggregated Tool Discovery (`tools/list`)
To discover all callable tools across all connected servers:
```bash
curl -X POST "http://localhost:8080/mcp" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list",
"params": {}
}'
```
#### Response with Namespaced Prefixes
The gateway prefixes each tool name with its server slug (`{serverSlug}__{toolName}`) to prevent collisions:
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"tools": [
{
"name": "postgres-db__run_query",
"description": "Execute read-only SQL queries on the replica database.",
"inputSchema": {
"type": "object",
"properties": {
"sql": { "type": "string" }
},
"required": ["sql"]
}
},
{
"name": "github-ops__create_pull_request",
"description": "Create a new pull request in repository.",
"inputSchema": {
"type": "object",
"properties": {
"title": { "type": "string" },
"branch": { "type": "string" }
},
"required": ["title", "branch"]
}
}
]
}
}
```
### Aggregated Tool Execution (`tools/call`)
Execute an aggregated tool by referencing its prefixed name:
```bash
curl -X POST "http://localhost:8080/mcp" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 2,
"method": "tools/call",
"params": {
"name": "postgres-db__run_query",
"arguments": {
"sql": "SELECT id, email, created_at FROM users LIMIT 5;"
}
}
}'
```
The gateway extracts the prefix `postgres-db`, resolves the server credentials and network transport, and dispatches the call to the target server.
---
## 2. Scoped Server Routing (`POST /mcp/{slug}`)
When an agent needs to communicate exclusively with one server (such as an internal Jira or Kubernetes MCP server), use the scoped endpoint:
```bash
curl -X POST "http://localhost:8080/mcp/github-ops" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list",
"params": {}
}'
```
In scoped mode:
- Tool names are presented **unprefixed** (e.g., `create_pull_request` rather than `github-ops__create_pull_request`).
- Execution is isolated strictly to the target server, reducing fan-out overhead.
---
## 3. Curated Toolsets (`POST /mcp/toolsets/{slug}`)
A **Toolset** is a curated collection of specific tools drawn from one or more MCP servers. Toolsets allow organizations to enforce principle-of-least-privilege access:
- Example: An `analyst-toolset` might expose `postgres-db__run_query` and `slack__post_message`, but hide `postgres-db__drop_table` and `aws__terminate_instance`.
```bash
curl -X POST "http://localhost:8080/mcp/toolsets/analyst-toolset" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list",
"params": {}
}'
```
---
## 4. Plain REST Bridge (`/mcp-rest/*`)
For client applications, web frontends, or microservices that prefer standard HTTP REST semantics over JSON-RPC 2.0 envelopes, the gateway provides the **Plain REST Bridge**.
### List Tools (`GET /mcp-rest/tools/list`)
Retrieve available tools as a clean JSON array:
```bash
# List all tools in organization
curl -X GET "http://localhost:8080/mcp-rest/tools/list" \
-H "Authorization: Bearer sk-gw-client-key"
# Filter tools to a specific server
curl -X GET "http://localhost:8080/mcp-rest/tools/list?server_slug=postgres-db" \
-H "Authorization: Bearer sk-gw-client-key"
```
#### Response
```json
{
"tools": [
{
"id": "tool_019283",
"server_id": "srv_pg_prod",
"name": "run_query",
"description": "Execute read-only SQL queries on the replica database.",
"input_schema": {
"type": "object",
"properties": {
"sql": { "type": "string" }
},
"required": ["sql"]
}
}
]
}
```
### Call Tool (`POST /mcp-rest/tools/call`)
Invoke a tool with a standard JSON body without JSON-RPC version headers or ID matching:
```bash
curl -X POST "http://localhost:8080/mcp-rest/tools/call" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"server_slug": "postgres-db",
"tool_name": "run_query",
"arguments": {
"sql": "SELECT COUNT(*) FROM orders WHERE status = 'pending';"
}
}'
```
#### Successful Execution Response
```json
{
"content": [
{
"type": "text",
"text": "{\"count\": 42}"
}
],
"is_error": false
}
```
### Ambiguous Tool Name Protection
If a caller provides `tool_name` without specifying `server_slug`, and that tool name exists across multiple connected MCP servers, the REST bridge safely rejects the call with HTTP 400 (`AmbiguousToolName`) rather than executing unpredictably:
```json
{
"error": {
"code": "AmbiguousToolName",
"message": "Tool name \"run_query\" exists on multiple servers; server_slug is required.",
"tool_name": "run_query",
"matching_servers": ["postgres-db", "snowflake-warehouse"]
}
}
```
> [!TIP]
> Use the Plain REST Bridge when building internal tools, web dashboards, or microservices that need to call MCP tools via standard `fetch()`, Axios, or Go `http.Client` without pulling in complex JSON-RPC dependencies.
---
## MCP Progressive Tool Discovery
URL: https://goroute-ai.github.io/docs/agents/mcp-progressive-discovery
Markdown: https://goroute-ai.github.io/docs/agents/mcp-progressive-discovery.md
Section: Control Plane Documentation > Connect agents and tools
# MCP Progressive Tool Discovery
In enterprise AI deployments, organizations frequently integrate dozens of Model Context Protocol (MCP) servers and external APIs, resulting in catalogs spanning hundreds or thousands of callable tools.
Directly serializing every tool's complete JSON Schema into an LLM's system prompt introduces severe operational bottlenecks:
- **Massive Context Consumption**: 100+ complete tool definitions consume between 20,000 to 50,000 input tokens per inference request.
- **FinOps Inefficiencies**: Sending massive schemas on every chat turn exponentially inflates API costs.
- **Model Attention Degradation**: Overly long system prompts impair the model's ability to focus on user instructions and reasoning steps.
- **Security & Poisoning Risks**: Exposing internal database structures and schemas upfront increases exposure to indirect prompt injection attacks.
GoRoute AI Gateway solves this challenge through **MCP Progressive Tool Discovery**, a token-efficient architecture using compact 256-bit opaque handles (`tool_hdl_...`), an enterprise registry feed (`GET /v1/goroute/mcp/registry`), schema hydration meta-tools, and governed OpenAPI 3.0/3.1 transport.
---
## 1. Architectural Overview
```mermaid
sequenceDiagram
participant User as Client / Agent
participant GW as GoRoute AI Gateway
participant Redis as Redis Fabric (L2)
participant Upstream as Upstream MCP Server
User->>GW: 1. GET /v1/goroute/mcp/registry
GW-->>User: 2. Lightweight Feed (Name & tool_hdl)
Note over User: Agent selects required tool
User->>GW: 3. POST /v1/chat/completions (search_tools)
GW-->>User: 4. Compact Matches with Opaque Handles
User->>GW: 5. POST /mcp (get_tool_schema)
GW->>Redis: 6. Resolve Handle & Verify Hash
Redis-->>GW: 7. Verified Handle Record
GW-->>User: 8. Full Input JSON Schema
User->>GW: 9. POST /mcp (Execute tool call)
GW->>Upstream: 10. Forward Tool Invocation
Upstream-->>GW: 11. Tool Result Payload
GW-->>User: 12. Execution Response
```
Instead of sending full schemas upfront, the gateway supplies the LLM with two lightweight meta-tools (`search_tools` and `get_tool_schema`), allowing the agent to discover and hydrate only the exact schemas required for the immediate task.
---
## 2. 256-Bit Opaque Tool Handles (`tool_hdl_...`)
When tools are indexed by the gateway, the `ProgressiveDiscoveryManager` generates a cryptographically random 256-bit handle with the `tool_hdl_` prefix.
### Handle Generation (`internal/mcp/progressive_discovery.go`)
```go
const (
HandlePrefix = "tool_hdl_"
HandleTTL = 1 * time.Hour
RedisHdlPrefix = "gw:mcp:hdl:"
)
// ToolHandleRecord stores the binding between a 256-bit opaque handle and a verified tool.
type ToolHandleRecord struct {
Handle string `json:"handle"`
ToolID string `json:"tool_id"`
ServerID string `json:"server_id"`
ServerSlug string `json:"server_slug"`
ToolName string `json:"tool_name"`
SchemaHash string `json:"schema_hash"`
OrgID string `json:"org_id"`
CreatedAt time.Time `json:"created_at"`
ExpiresAt time.Time `json:"expires_at"`
}
```
Each handle is backed by 32 bytes (256 bits) of cryptographically secure random entropy (`crypto/rand`), yielding a 64-character hex string:
```text
tool_hdl_a4f91c0e38b29d47c6e11894a3bc76d892015fa43d67bc1e89e02345fa6712bc
```
### Fail-Closed Freshness Verification
Handles have a 1-hour TTL (`HandleTTL = 1 * time.Hour`) and are cached in Redis under `gw:mcp:hdl:{handle}`.
When an agent presents a handle to hydrate a schema, the gateway performs strict cryptographic freshness verification:
1. The gateway looks up the `ToolHandleRecord` in Redis or in-memory fallback.
2. The gateway retrieves the current tool definition from the repository database.
3. The gateway compares the tool's SHA-256 schema hash (`tool.ToolDefHash`) with the recorded hash in the handle (`record.SchemaHash`).
4. If an upstream administrator has updated or modified the tool schema during the session, the gateway rejects the request with `ErrStaleToolHandle`:
```go
if tool.ToolDefHash != record.SchemaHash {
return nil, nil, fmt.Errorf("%w: handle schema hash %s does not match repository hash %s",
ErrStaleToolHandle, record.SchemaHash, tool.ToolDefHash)
}
```
> [!IMPORTANT]
> Fail-closed freshness verification prevents race conditions and schema injection: an agent cannot execute a cached tool if the underlying schema or permissions have been altered while the handle was in flight.
---
## 3. Catalog Registry Feed (`GET /v1/goroute/mcp/registry`)
Client applications and autonomous frameworks can retrieve the entire registered tool catalog without schemas using the registry feed endpoint:
```bash
curl -X GET "http://localhost:8080/v1/goroute/mcp/registry" \
-H "Authorization: Bearer sk-gw-client-key-production"
```
### Response Format
```json
[
{
"server_slug": "postgres-service",
"server_name": "Production Read-Replica DB",
"transport_type": "streamable_http",
"protocol_version": "2026-07-28",
"endpoint_url": "https://mcp-pg.internal.acme.com/mcp",
"tool_count": 2,
"tools": [
{
"name": "run_read_query",
"server_slug": "postgres-service",
"description": "Executes read-only SQL queries against the analytics database.",
"tool_handle": "tool_hdl_8f93a1c0d45b7e82910fa32456bc718293041fa982345bc78129034fa9812bc4"
},
{
"name": "explain_query_plan",
"server_slug": "postgres-service",
"description": "Generates query execution plans without running mutations.",
"tool_handle": "tool_hdl_1029384756afbecd0192837465fabcde0192837465afbecd0192837465afbecd"
}
]
},
{
"server_slug": "github-actions",
"server_name": "CI/CD Pipeline Automation",
"transport_type": "openapi",
"protocol_version": "2026-07-28",
"endpoint_url": "https://api.github.com",
"tool_count": 1,
"tools": [
{
"name": "trigger_workflow",
"server_slug": "github-actions",
"description": "Triggers automated test and deployment workflow runs.",
"tool_handle": "tool_hdl_778899aabbccddeeff00112233445566778899aabbccddeeff00112233445566"
}
]
}
]
```
---
## 4. Schema Hydration Meta-Tools
When connecting an LLM to the gateway via progressive discovery, the gateway exposes two core meta-tools:
### Meta-Tool 1: `search_tools`
Allows the agent to search for tools based on intent or query terms:
```json
{
"name": "search_tools",
"description": "Searches the enterprise tool catalog for matching capabilities and returns lightweight tool summaries with opaque handles.",
"parameters": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "Keyword search query (e.g. 'postgres', 'sql', 'deploy', 'analytics')"
},
"limit": {
"type": "integer",
"description": "Maximum number of results to return (default 10, max 50)"
}
},
"required": ["query"]
}
}
```
### Meta-Tool 2: `get_tool_schema`
Fetches the complete JSON Schema definition for a specific tool using its opaque handle:
```json
{
"name": "get_tool_schema",
"description": "Hydrates and returns the authoritative JSON inputSchema for a specific tool using its 256-bit opaque handle.",
"parameters": {
"type": "object",
"properties": {
"tool_handle": {
"type": "string",
"description": "The opaque handle starting with 'tool_hdl_' obtained from search_tools or the registry feed."
}
},
"required": ["tool_handle"]
}
}
```
### Schema Hydration Response Example
When the LLM invokes `get_tool_schema` with handle `tool_hdl_8f93a1c0d45b7e82910fa32456bc718293041fa982345bc78129034fa9812bc4`, the gateway responds:
```json
{
"name": "run_read_query",
"server_slug": "postgres-service",
"description": "Executes read-only SQL queries against the analytics database.",
"inputSchema": {
"type": "object",
"properties": {
"sql": {
"type": "string",
"description": "The SQL SELECT statement to execute."
},
"max_rows": {
"type": "integer",
"description": "Maximum rows to return.",
"default": 100
}
},
"required": ["sql"]
}
}
```
Now, the agent holds only the single schema it actually needs, preserving thousands of context tokens.
---
## 5. Governed OpenAPI 3.0 / 3.1 Transport
GoRoute AI Gateway can ingest any standard OpenAPI 3.0 or 3.1 REST API specification and automatically project it as an MCP tool server.
### Automatic Tool Synthesis (`internal/mcp/transport_openapi.go`)
The gateway's `OpenAPITransport` parses operations (`paths`), extracts input schemas from `parameters` and `requestBody`, and synthesizes native MCP tool schemas:
```go
type OpenAPIOperation struct {
OperationID string `json:"operation_id"`
Method string `json:"method"`
PathPattern string `json:"path_pattern"`
Summary string `json:"summary"`
Description string `json:"description"`
InputSchema map[string]any `json:"input_schema"`
PathParams []string `json:"path_params"`
QueryParams []string `json:"query_params"`
}
```
### Features of Governed OpenAPI Transport:
- **Path Parameter Interpolation**: Automatically maps tool arguments to URL path variables (e.g. `/v1/users/{userId}` $\rightarrow$ substituted with `args["userId"]`).
- **Query Parameter Mapping**: Arguments designated as query parameters are encoded into the URL query string.
- **JSON Request Body Encoding**: Body parameters are validated against the OpenAPI schema and posted as `application/json`.
- **SSRF Hardening**: Outbound HTTP calls use the gateway's centralized `network.Policy` with IP pinning, loopback filtering, and cloud metadata blocking.
- **Arbitrary Decimal Handling**: Monetary amounts and high-precision IDs in API responses are preserved using `json.Number` without IEEE-754 precision loss.
---
## 6. Token Savings Comparison
| Architecture | 50 Tools | 200 Tools | 500 Tools |
| :--- | :--- | :--- | :--- |
| **Traditional Upfront Ingestion** | ~15,000 tokens | ~60,000 tokens | Exceeds context limits / ~150,000 tokens |
| **Progressive Tool Discovery** | ~450 tokens *(meta-tools + handles)* | ~500 tokens | ~550 tokens |
| **Net Token Reduction** | **97.0%** | **99.1%** | **99.6%** |
> [!TIP]
> Use progressive discovery for multi-agent workflows (such as LangChain, AutoGen, or Claude Code) that interact with corporate microservice meshes. The agent retains full visibility into available tools while avoiding token exhaustion.
---
## MCP Stdio Process Sandboxing
URL: https://goroute-ai.github.io/docs/agents/mcp-stdio-sandboxing
Markdown: https://goroute-ai.github.io/docs/agents/mcp-stdio-sandboxing.md
Section: Control Plane Documentation > Connect agents and tools
# MCP Stdio Process Sandboxing
Local command-line MCP tools (such as CLI database clients, git helpers, file system scanners, and Python analysis scripts) communicate over standard input and output streams (`stdin` / `stdout`).
Executing untrusted or arbitrary binaries on a production host presents critical security risks:
- **Environment Leakage**: Malicious child processes could inspect parent environment variables and exfiltrate database credentials, cloud access keys, or the gateway's AES-256 master keys.
- **Resource Exhaustion**: Hung or runaway scripts could consume 100% CPU, leak file descriptors, or flood stdout to exhaust system memory.
- **Arbitrary Code Execution**: Attackers attempting command injection could spawn unauthorized shells (`/bin/sh`, `/bin/bash`, `nc`).
GoRoute AI Gateway embeds a secure **Process Supervisor** (`internal/mcp/transport_stdio.go`) that isolates Stdio tools inside strict execution sandboxes with fail-closed access controls.
```mermaid
flowchart TD
Req["Gateway Tool Invocation"] --> CheckAllow{"Binary in AllowedBinaries?"}
CheckAllow -->|No| Reject["403 ErrBinaryNotAllowed
(Fail-Closed)"]
CheckAllow -->|Yes| Sandbox["Spawn Subprocess Sandbox"]
subgraph Subprocess["Isolated Child Process"]
Env["Purged Environment
PATH, HOME, GOROUTE_SANDBOX=1
(Master keys stripped)"]
Timeout["Context Deadline (30s)
(Auto SIGKILL on timeout)"]
Buffer["Output Buffer Limit (10MB)
(io.LimitReader)"]
Stderr["Background Stderr Drain"]
end
Sandbox --> Subprocess
Subprocess --> Res["Parse Line-Delimited JSON-RPC Response"]
```
---
## 1. Security Architecture & Invariants
### 1. Fail-Closed Binary Allowlists
The supervisor will refuse to spawn any binary unless it explicitly appears in `StdioSecurityPolicy.AllowedBinaries`.
```go
// StdioSecurityPolicy defines the sandboxing rules for subprocess execution.
type StdioSecurityPolicy struct {
AllowedBinaries []string `json:"allowed_binaries"`
DefaultTimeout time.Duration `json:"default_timeout"`
MaxOutputBytes int64 `json:"max_output_bytes"`
}
```
The gateway checks both the absolute filesystem path and the executable base name (`filepath.Base`):
```go
func (t *StdioTransport) isBinaryAllowed(binaryPath string) bool {
if len(t.policy.AllowedBinaries) == 0 {
return false // Fail closed if allowlist is empty
}
base := filepath.Base(binaryPath)
for _, allowed := range t.policy.AllowedBinaries {
if allowed == "*" || allowed == binaryPath || allowed == base {
return true
}
}
return false
}
```
If an unauthorized binary is requested, execution halts immediately with `ErrBinaryNotAllowed`:
```text
mcp: stdio binary is not in approved allowlist: "/bin/sh"
```
### 2. Strict Environment Variable Scrubbing
When launching child processes, the gateway completely purges the host process environment. Secrets such as `GATEWAY_MASTER_ADMIN_KEY`, `GATEWAY_MASTER_ENCRYPTION_KEY`, and `GATEWAY_POSTGRES_DSN` are never inherited by child processes:
```go
// Scrub environment variables: strictly isolate process from gateway secrets
cmd.Env = []string{
"PATH=" + os.Getenv("PATH"),
"HOME=" + os.Getenv("HOME"),
"LANG=en_US.UTF-8",
"GOROUTE_SANDBOX=1",
}
// Inject only explicitly declared static headers as environment variables
for k, v := range server.StaticHeaders {
safeKey := strings.ToUpper(strings.ReplaceAll(k, "-", "_"))
cmd.Env = append(cmd.Env, safeKey+"="+v)
}
```
### 3. Execution Deadlines & Resource Quotas
- **Default Timeout (30s)**: Every tool invocation is governed by `context.WithTimeout(ctx, policy.DefaultTimeout)`. If a process hangs, enters an infinite loop, or waits for user input, the supervisor terminates it immediately with `SIGKILL`.
- **Output Buffer Clamping (10 MB)**: Stdout streams are bounded via `io.LimitReader(stdout, policy.MaxOutputBytes)`. A runaway process attempting to flood the output buffer is truncated, preventing out-of-memory panics.
- **Background Stderr Diagnostics**: Stderr is drained in an independent background goroutine. If a subprocess crashes with a non-zero exit code, its stderr diagnostics are captured in gateway logs to assist operators without polluting JSON-RPC stdout channels.
---
## 2. Configuration (`configs/config.yaml`)
To enable and configure Stdio execution in your deployment:
```yaml
mcp:
enabled: true
# Explicitly enable local subprocess execution (defaults to false)
allow_stdio_servers: true
execution_timeout_seconds: 30
max_payload_bytes: 10485760 # 10 MB
# Permitted executable allowlist
allowed_binaries:
- "/usr/local/bin/git"
- "sqlite3"
- "node"
- "python3"
```
---
## 3. Registering a Stdio MCP Server via Admin API
Register a local CLI tool using the `stdio://` URL scheme:
```bash
curl -X POST "http://localhost:8080/admin/v1/mcp-servers" \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"slug": "local-sqlite",
"name": "Local SQLite Query Tool",
"transport_type": "stdio",
"protocol_version": "2026-07-28",
"endpoint_url": "stdio://sqlite3 /data/production.db",
"is_active": true,
"static_headers": {
"SQLITE_ENABLE_JSON1": "1"
}
}'
```
### Argument Parsing & Quotes
The gateway's command parser (`parseCommandLine`) handles quoted arguments and backslash escapes cleanly:
```text
stdio://python3 /opt/tools/analyzer.py --mode="read-only" --db="sales report.db"
```
---
## 4. Operational Best Practices
> [!NOTE]
> In containerized Kubernetes deployments, ensure that any binaries referenced by Stdio servers are pre-installed in the container image or mounted via read-only volumes.
> [!CAUTION]
> Never add general-purpose command interpreters (such as `bash`, `sh`, `zsh`, `sudo`, or `eval`) to `allowed_binaries`. Restrict the allowlist strictly to dedicated CLI utilities with known, bounded execution profiles.
---
## Agent-to-Agent (A2A) Protocol Overview
URL: https://goroute-ai.github.io/docs/agents/a2a-overview
Markdown: https://goroute-ai.github.io/docs/agents/a2a-overview.md
Section: Control Plane Documentation > Connect agents and tools
# Agent-to-Agent (A2A) Protocol Overview
The **Linux Foundation Agent-to-Agent Protocol (A2A v1.0)** is an open, vendor-neutral specification governing autonomous multi-agent communication, collaborative task execution, and real-time event distribution.
GoRoute AI Gateway implements an enterprise-grade A2A router and messaging hub, enabling independent agent systems (e.g., LangChain agents, AutoGen swarms, Claude Code, and autonomous background bots) to coordinate tasks securely with enterprise auth, audit trails, and multi-tenant isolation.
```mermaid
flowchart TD
subgraph Clients["Client Agents"]
AgentA["Initiator Agent A"]
AgentB["Client Bot B"]
end
subgraph Gateway["GoRoute AI Gateway Hub"]
DualBinding["Dual Ingress Binding
(HTTP+JSON & JSON-RPC 2.0)"]
TaskRouter["A2A Task Router Service"]
StreamSub["Subscribe-First SSE Engine
(Monotonic Event Sequences)"]
Outbox["Transactional Outbox Worker
(DLQ & Exponential Backoff)"]
end
subgraph Upstream["Target Agent Network"]
TargetAgent["Autonomous Worker Agent C"]
WebhookSubscriber["Webhook Subscriber D"]
end
AgentA -->|HTTP POST or JSON-RPC| DualBinding
DualBinding --> TaskRouter
TaskRouter --> TargetAgent
TargetAgent -->|Events| TaskRouter
AgentB -->|GET :subscribe SSE| StreamSub
TaskRouter --> Outbox
Outbox -->|HTTP Webhook Push| WebhookSubscriber
```
---
## 1. Dual Protocol Binding: HTTP+JSON and JSON-RPC 2.0
To ensure compatibility across different software ecosystems, GoRoute AI Gateway exposes identical agent capabilities over two transport bindings:
### 1. HTTP+JSON REST Binding
Idiomatic REST APIs designed for standard HTTP microservices and web clients:
| Operation | Method & Route | Description |
| :--- | :--- | :--- |
| **Send Message** | `POST /a2a/v1/agents/{slug}/messages` | Submits a message to an agent, initializing or continuing a task. |
| **Get Task** | `GET /a2a/v1/agents/{slug}/tasks/{taskID}` | Retrieves current task status, artifacts, and execution state. |
| **List Tasks** | `GET /a2a/v1/agents/{slug}/tasks` | Lists active and historical tasks for the caller. |
| **Cancel Task** | `POST /a2a/v1/agents/{slug}/tasks/{taskID}:cancel` | Signals cancellation to an in-flight agent task. |
| **Subscribe Stream** | `GET /a2a/v1/agents/{slug}/tasks/{taskID}:subscribe` | Opens a real-time SSE stream of task events. |
### 2. JSON-RPC 2.0 Binding
Standardized RPC interface accessible on a single URL (`POST /a2a/v1/agents/{slug}`):
```bash
curl -X POST "http://localhost:8080/a2a/v1/agents/risk-analyst" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 101,
"method": "sendMessage",
"params": {
"message": {
"role": "user",
"content": "Evaluate quarterly exposure to semiconductor supply shocks."
}
}
}'
```
Supported JSON-RPC methods: `sendMessage`, `getTask`, `listTasks`, `cancelTask`, `createPushConfig`, `getPushConfig`, `deletePushConfig`.
---
## 2. Subscribe-First SSE Real-Time Streaming
A common vulnerability in distributed SSE streaming is the "race window": if a client queries the database for past events and then connects to the live stream, any event emitted between the database read and the socket connection is permanently lost.
GoRoute AI Gateway implements the **Subscribe-First SSE Protocol** (`internal/a2a/streams.go`):
```mermaid
sequenceDiagram
participant Client as Subscriber Client
participant GW as Gateway SSE Handler
participant Redis as Redis Pub/Sub
participant DB as PostgreSQL (a2a_events)
Client->>GW: 1. GET /tasks/{id}:subscribe?after_seq=4
Note over GW: 1. Subscribe to Redis FIRST
GW->>Redis: 2. Subscribe(gw:a2a:events:{id})
GW-->>Client: 3. HTTP 200 text/event-stream
Note over GW: 2. Backfill Historical Events
GW->>DB: 4. ListEvents(taskID, seq > 4)
DB-->>GW: 5. Historical Events (seq 5, 6)
GW-->>Client: 6. SSE id: 5, id: 6
Note over GW: 3. Gap Detection & Backfill
Redis-->>GW: 7. Live Event (seq 8) [Gap: seq 7 missing]
GW->>DB: 8. Fetch Missing Gap (seq 7)
DB-->>GW: 9. Event 7 Payload
GW-->>Client: 10. SSE id: 7
GW-->>Client: 11. SSE id: 8
```
### Key Stream Resiliency Features:
1. **Subscribe-First Sequencing**: Connects to the Redis event channel `gw:a2a:events:{taskID}` before fetching database snapshots, buffering live events in a channel.
2. **Monotonic Event Sequencing**: Every task event is assigned an auto-incrementing 64-bit integer sequence number (`id: 1`, `id: 2`, `id: 3`).
3. **Resumption via `Last-Event-ID`**: If a subscriber disconnects, it reconnects with standard `Last-Event-ID: ` or `?after_seq=`. The gateway replays all missed events seamlessly.
4. **Gap Detection & Deduplication**: If a live buffered event skips a sequence number, the gateway queries PostgreSQL to backfill the missing gap before delivering live events. Duplicate events already delivered during backfill are discarded.
5. **Periodic Keep-Alive**: Emits comment frames (`: keepalive\n\n`) every 15 seconds to prevent intermediate proxy timeout disconnects.
---
## 3. Transactional Outbox Pattern & Webhook Delivery
For asynchronous event delivery to external systems, the gateway implements a **Transactional Outbox Worker** (`internal/a2a/outbox_worker.go`).
```mermaid
flowchart LR
Task["Agent Task Mutation"] -->|Atomic Transaction| DB[("PostgreSQL
1. Task Event
2. a2a_outbox_events")]
subgraph OutboxWorker["Background Outbox Worker"]
Claim["Claim Pending Batches (50 events)"]
HTTPPost["HTTP POST with Idempotency-Key"]
RetryLogic{"Delivery Success?"}
end
DB --> Claim
Claim --> HTTPPost
HTTPPost --> RetryLogic
RetryLogic -->|Yes| MarkSent["Status: SENT"]
RetryLogic -->|"No (under 5 retries)"| Backoff["Exponential Backoff
100ms * 2^attempt"]
RetryLogic -->|"No (max retries exceeded)"| DLQ["Status: DEAD_LETTER (DLQ)"]
MarkSent --> DB
Backoff --> DB
DLQ --> DB
```
### Operational Guarantees:
- **Zero-Loss Transactions**: The task status mutation and the outbound webhook event are written within the same database transaction. Events can never be lost due to process crashes.
- **Idempotency & Attribution Headers**: Outbound webhook POSTs carry strict idempotency and audit metadata:
```http
POST /webhooks/agent-events HTTP/1.1
Host: client-service.acme.com
Content-Type: application/json
Idempotency-Key: evt_01928374-abcd-ef01
X-Gateway-Event-Id: evt_01928374-abcd-ef01
X-Gateway-Task-Id: task_98765432-feed-dead
X-Gateway-Delivery-Attempt: 1
```
- **Exponential Backoff**: Failed deliveries back off exponentially (`100ms * 2^attempt`).
- **Dead-Letter Queue (DLQ)**: After 5 consecutive failures (`a2a.max_push_retries: 5`), the event status transitions to `dead_letter` for operator inspection.
- **Emergency Kill-Switch**: Webhook delivery can be deactivated instantly per agent or globally by deleting or toggling push notification configs.
---
## Cryptographic Agent Cards (JCS & JWS)
URL: https://goroute-ai.github.io/docs/agents/a2a-agent-cards
Markdown: https://goroute-ai.github.io/docs/agents/a2a-agent-cards.md
Section: Control Plane Documentation > Connect agents and tools
# Cryptographic Agent Cards (JCS & JWS)
In autonomous multi-agent networks, agents must be able to discover each other's capabilities dynamically while verifying that an agent's declared schemas, endpoints, and credentials have not been tampered with in transit.
Under the Linux Foundation A2A specification, each autonomous agent publishes an **Agent Card** at a well-known URL. GoRoute AI Gateway acts as a secure identity proxy, enforcing **RFC 8785 JSON Canonicalization Scheme (JCS)** and **RFC 7515 JSON Web Signature (JWS)** signing and verification.
```mermaid
flowchart LR
UpstreamAgent["Upstream Agent
(Publishes Agent Card)"] -->|Raw JSON| Gateway["GoRoute AI Gateway Signer"]
subgraph CryptoPipeline["Cryptographic Pipeline"]
JCS["RFC 8785 JCS Canonicalization
(Deterministic Lexicographical Order)"]
Digest["SHA-256 Digest"]
JWS["RFC 7515 JWS Signing
(ES256, RS256, EdDSA)"]
end
Gateway --> JCS
JCS --> Digest
Digest --> JWS
JWS --> PublicCard["GET /.well-known/agent-card.json
(Signed Agent Card)"]
```
---
## 1. Discovering the Agent Card (`GET /.well-known/agent-card.json`)
Clients and external agent swarms discover agent capabilities via the well-known discovery route:
```bash
# Discover default gateway agent card
curl -X GET "http://localhost:8080/.well-known/agent-card.json"
# Discover a specific registered agent by slug
curl -X GET "http://localhost:8080/.well-known/agent-card.json?agent=risk-analyst"
```
### Agent Card JSON Schema Example
```json
{
"name": "Risk Analysis Agent",
"description": "Autonomous financial risk evaluation and stress-testing agent.",
"version": "1.0.0",
"supportedInterfaces": [
{
"protocolBinding": "HTTP+JSON",
"url": "http://localhost:8080/a2a/v1/agents/risk-analyst"
},
{
"protocolBinding": "JSON-RPC",
"url": "http://localhost:8080/a2a/v1/agents/risk-analyst"
},
{
"protocolBinding": "GRPC",
"url": "localhost:9090"
}
],
"capabilities": {
"streaming": true,
"pushNotifications": true,
"statePersistence": true
},
"signature": {
"alg": "ES256",
"kid": "gw-a2a-signer-v1",
"jws": "eyJhbGciOiJFUzI1NiIsImtpZCI6Imd3LWEyYS1zaWduZXItdjEifQ.eyJjYXBhYmlsaXRpZXMi...signature"
}
}
```
---
## 2. Deterministic Serialization: RFC 8785 JCS
Standard JSON encoders do not guarantee deterministic serialization: key ordering is arbitrary, whitespace varies, and numeric representations differ across platforms. Without canonical serialization, cryptographic signatures fail.
GoRoute AI Gateway implements **RFC 8785 JSON Canonicalization Scheme (JCS)** (`internal/a2a/card.go`):
1. **Lexicographical Key Sorting**: Object member names are sorted lexicographically by Unicode code point values.
2. **Whitespace Stripping**: Eliminates all non-essential formatting whitespace outside string literals.
3. **Canonical Number Formatting**: Integers and arbitrary-precision floats are represented without exponents when within safe ranges, preserving decimal precision.
4. **UTF-8 Character Escaping**: Only essential control characters (`"`, `\`) are escaped.
```go
// CanonicalizeJCS serializes any JSON-compatible object into RFC 8785 JCS.
func CanonicalizeJCS(v any) ([]byte, error) {
raw, err := json.Marshal(v)
if err != nil {
return nil, err
}
var parsed any
dec := json.NewDecoder(bytes.NewReader(raw))
dec.UseNumber()
if err := dec.Decode(&parsed); err != nil {
return nil, err
}
return canonicalizeValue(parsed)
}
```
---
## 3. Cryptographic Signing: RFC 7515 JWS
Once canonicalized, the card payload is signed according to **RFC 7515 JSON Web Signature (JWS)**.
### Supported Cryptographic Algorithms
The gateway supports three standard asymmetric signing algorithms configured in `configs/config.yaml`:
- **`ES256`** (Default): ECDSA with the NIST P-256 curve and SHA-256. Recommended for modern high-performance cloud environments.
- **`RS256`**: RSA signature with SHA-256 (minimum 2048-bit key).
- **`EdDSA`**: Ed25519 high-speed elliptic curve signature scheme.
### Compact JWS Serialization
The signature is formatted in compact 3-part Base64URL notation:
```text
BASE64URL(UTF8(ProtectedHeader)) . BASE64URL(JCS(Payload)) . BASE64URL(SignatureBytes)
```
Where the Protected Header declares the algorithm and key ID:
```json
{
"alg": "ES256",
"kid": "gw-a2a-signer-v1"
}
```
---
## 4. Ingress Gateway URL Rewriting & Re-Signing
When GoRoute AI Gateway proxies an external or upstream A2A agent:
1. **Upstream Ingestion**: The gateway fetches the upstream agent card and validates its cryptographic signature against the upstream's published JWKS.
2. **Endpoint Rewriting**: Upstream interface URLs (e.g. `http://internal-agent-host:3000`) are rewritten to point to the gateway's unified ingress proxy (`https://gateway.acme.com/a2a/v1/agents/{slug}`).
3. **Re-Signing**: Because rewriting modifies the payload, the gateway strips the old signature, canonicalizes the updated payload via RFC 8785 JCS, and re-signs the document using the gateway's private key (`a2a.signing_key_id`).
This guarantees that downstream clients can verify the authenticity of the card while directing all live traffic through the gateway's rate limiters, FinOps ledgers, and security guardrails.
---
## 5. Verification Configuration (`configs/config.yaml`)
```yaml
a2a:
enabled: true
signing_key_id: "gw-a2a-signer-v1"
allowed_signing_algs:
- "ES256"
- "RS256"
- "EdDSA"
# Upstream trusted agent public JWKS keys
trusted_card_jwks:
"risk-analyst": "https://risk-agent.partner.com/.well-known/jwks.json"
"compliance-bot": "https://compliance.internal.bank/.well-known/jwks.json"
```
> [!IMPORTANT]
> In production environments, never set `a2a.allow_legacy_card_discovery: true`. All agent cards must be cryptographically signed and verifiable against approved keys to prevent man-in-the-middle capability spoofing.
---
## A2A OpenAI Bridge & gRPC Service
URL: https://goroute-ai.github.io/docs/agents/a2a-bridge-and-grpc
Markdown: https://goroute-ai.github.io/docs/agents/a2a-bridge-and-grpc.md
Section: Control Plane Documentation > Connect agents and tools
# A2A OpenAI Bridge & gRPC Service
GoRoute AI Gateway enables seamless interoperability between autonomous A2A agents and existing AI application ecosystems:
1. **OpenAI Compatibility Bridge (`model: "a2a/{slug}"`)**: Exposes autonomous agents over standard `POST /v1/chat/completions`, allowing off-the-shelf OpenAI SDKs, LangChain, Cursor, Claude Code, and OpenWebUI to call A2A agents without custom protocol libraries.
2. **Native gRPC Service (`lf.a2a.v1.A2AService`)**: High-performance, low-latency Protobuf RPCs with server-side streaming designed for production microservice architectures.
```mermaid
flowchart TD
subgraph Clients["Diverse Client Ecosystem"]
OpenAISDK["OpenAI Python / Node SDK
(model: 'a2a/risk-analyst')"]
LangChain["LangChain / LlamaIndex"]
GRPCClient["Internal Go / Java Microservice
(Native gRPC Client)"]
end
subgraph Gateway["GoRoute AI Gateway"]
OpenAIBridge["OpenAI Bridge Handler
(internal/a2a/openai_bridge.go)"]
GRPCServer["A2A gRPC Server
(lf.a2a.v1.A2AService)"]
RBAC["4-Layer RBAC Engine"]
TaskRouter["A2A Router Service"]
end
subgraph Agents["Autonomous Agents"]
AgentWorker["Risk Analyst Agent Worker"]
end
OpenAISDK -->|POST /v1/chat/completions| OpenAIBridge
LangChain -->|POST /v1/chat/completions| OpenAIBridge
GRPCClient -->|gRPC Channel: 9090| GRPCServer
OpenAIBridge --> RBAC
GRPCServer --> RBAC
RBAC --> TaskRouter
TaskRouter --> AgentWorker
```
---
## 1. OpenAI Compatibility Bridge (`model: "a2a/{slug}"`)
The OpenAI Compatibility Bridge translates standard Chat Completions requests into A2A agent tasks, preserving conversational state, streaming SSE chunks, and enforcing tenant permissions.
### Invoking an Agent via cURL
To invoke an agent named `risk-analyst`, simply set `model` to `"a2a/risk-analyst"`:
```bash
curl -X POST "http://localhost:8080/v1/chat/completions" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "Content-Type: application/json" \
-d '{
"model": "a2a/risk-analyst",
"messages": [
{
"role": "user",
"content": "Perform sensitivity analysis on portfolio beta under a 200bps rate hike."
}
],
"stream": false
}'
```
#### Response Format
The gateway formats the agent's task output into standard OpenAI response JSON:
```json
{
"id": "chatcmpl_a2a_task_01928374",
"object": "chat.completion",
"created": 1731340800,
"model": "a2a/risk-analyst",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "### Portfolio Sensitivity Analysis\nUnder a 200bps rate increase, duration-sensitive fixed income assets experience an estimated 4.2% drawdown while high-dividend equity beta contracts by 0.15..."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 38,
"completion_tokens": 154,
"total_tokens": 192
}
}
```
### Python OpenAI SDK Integration
Existing code using the official `openai` Python package works without code changes:
```python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="sk-gw-client-key"
)
# Call the autonomous agent directly
response = client.chat.completions.create(
model="a2a/risk-analyst",
messages=[
{"role": "user", "content": "Analyze supply chain risks for our automotive supplier portfolio."}
],
stream=True
)
for chunk in response:
content = chunk.choices[0].delta.content or ""
print(content, end="", flush=True)
```
### Multi-Turn Resumption & Context Tracking
To continue an existing agent task across multiple turns, pass the task identifier via HTTP headers or metadata:
```bash
curl -X POST "http://localhost:8080/v1/chat/completions" \
-H "Authorization: Bearer sk-gw-client-key" \
-H "X-Gateway-Task-Id: task_01928374-abcd-ef01" \
-H "Content-Type: application/json" \
-d '{
"model": "a2a/risk-analyst",
"messages": [
{"role": "user", "content": "Re-run the previous stress test assuming high inflation persistency."}
]
}'
```
---
## 2. Native gRPC Service (`lf.a2a.v1.A2AService`)
For backend systems, microservices, and Kubernetes clusters requiring high-throughput binary serialization, GoRoute AI Gateway exposes the native Linux Foundation A2A gRPC service.
### Protobuf Service Definition
```protobuf
syntax = "proto3";
package lf.a2a.v1;
service A2AService {
// Dispatches a message to an agent task
rpc SendMessage(SendMessageRequest) returns (SendMessageResponse);
// Queries task status, current state, and artifacts
rpc GetTask(GetTaskRequest) returns (Task);
// Lists active tasks for the authenticated caller
rpc ListTasks(ListTasksRequest) returns (ListTasksResponse);
// Signals cancellation to an in-flight agent task
rpc CancelTask(CancelTaskRequest) returns (Task);
// Real-time server-streaming of task progress events
rpc SubscribeToTask(SubscribeToTaskRequest) returns (stream StreamResponse);
// Webhook push notification configurations
rpc CreateTaskPushNotificationConfig(TaskPushNotificationConfig) returns (TaskPushNotificationConfig);
rpc GetTaskPushNotificationConfig(GetTaskPushNotificationConfigRequest) returns (TaskPushNotificationConfig);
rpc DeleteTaskPushNotificationConfig(DeleteTaskPushNotificationConfigRequest) returns (google.protobuf.Empty);
}
```
### gRPC Authentication & Metadata
gRPC clients authenticate by including the standard VirtualKey Bearer token in the gRPC incoming metadata context:
```go
md := metadata.Pairs(
"authorization", "Bearer sk-gw-client-key-prod",
"agent-slug", "risk-analyst",
)
ctx := metadata.NewOutgoingContext(context.Background(), md)
```
The gateway extracts the token, executes the 4-layer RBAC policy check, and attributes compute spend to the associated organization and team.
### Server-Streaming Task Subscription Example (Go)
```go
package main
import (
"context"
"fmt"
"io"
"log"
"github.com/goroute-ai/ai-gateway-go/internal/a2a/pb"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/metadata"
)
func main() {
conn, err := grpc.Dial("localhost:9090", grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
client := pb.NewA2AServiceClient(conn)
ctx := metadata.NewOutgoingContext(context.Background(), metadata.Pairs(
"authorization", "Bearer sk-gw-client-key-prod",
))
stream, err := client.SubscribeToTask(ctx, &pb.SubscribeToTaskRequest{
TaskId: "task_01928374",
AfterSeq: 0,
})
if err != nil {
log.Fatalf("subscribe failed: %v", err)
}
for {
event, err := stream.Recv()
if err == io.EOF {
break
}
if err != nil {
log.Fatalf("stream error: %v", err)
}
fmt.Printf("[Seq %d] Event: %s, Payload: %s\n",
event.EventSeq, event.EventType, event.PayloadJson)
}
}
```
> [!TIP]
> Use `lf.a2a.v1.A2AService` when building backend services that require sub-millisecond serialization overhead and high-volume parallel event streaming.
---
## Integrations Overview
URL: https://goroute-ai.github.io/docs/integrations/overview
Markdown: https://goroute-ai.github.io/docs/integrations/overview.md
Section: Control Plane Documentation > Connect your tools
# Integrations Overview
You can connect many existing SDKs, coding tools, and agent frameworks to GoRoute by changing the endpoint and credential they use. Most integrations do not need a new application flow. Choose the guide for your client below, then use the protocol details when you need to troubleshoot a connection.
GoRoute supports OpenAI-compatible (`/v1/*`, `/openai/v1/*`) and Anthropic-compatible (`/anthropic/v1/*`, `/v1/messages`) endpoints, along with MCP (`/mcp*`) and A2A (`/a2a/v1/*`) protocols.
---
## Base URLs by Protocol
| Client Ecosystem | Base URL | Supported Endpoints | Reference |
| :--- | :--- | :--- | :--- |
| **OpenAI-Compatible** | `http://localhost:8080/v1` | `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/responses`, `/v1/realtime`, `/v1/models`, `/v1/audio/*`, `/v1/images/*`, `/v1/files`, `/v1/batches`, `/v1/rerank`, `/v1/moderations` | [Gateway API](https://goroute-ai.github.io/docs/api/gateway-api) |
| **OpenAI Direct Alias** | `http://localhost:8080/openai/v1` | `/openai/v1/responses`, `/openai/v1/realtime` | [Responses API](https://goroute-ai.github.io/docs/gateway/endpoints/responses) |
| **Anthropic-Compatible** | `http://localhost:8080/anthropic` | `/anthropic/v1/messages`, `/anthropic/v1/messages/count_tokens` (also mounted at `/v1/messages` and `/v1/messages/count_tokens`) | [Anthropic Messages](https://goroute-ai.github.io/docs/gateway/endpoints/messages) |
| **Model Context Protocol (MCP)** | `http://localhost:8080/mcp` | `/mcp`, `/mcp/{serverSlug}`, `/mcp/toolsets/{toolsetSlug}`, `/mcp-rest/tools/*` | [MCP Overview](https://goroute-ai.github.io/docs/agents/mcp-overview) |
| **Agent-to-Agent (A2A)** | `http://localhost:8080/a2a/v1` | `/.well-known/agent-card.json`, `/a2a/v1/agents/{agentSlug}/`, `/a2a/v1/agents/{agentSlug}/tasks*` | [A2A Overview](https://goroute-ai.github.io/docs/agents/a2a-overview) |
---
## Authentication with Virtual Keys (`sk-gw-...`)
Data-plane routes authenticate callers using **Virtual Keys** prefixed with `sk-gw-` (issued via the Operator UI at `/ui/` or `POST /admin/v1/keys`). See [Virtual Keys](https://goroute-ai.github.io/docs/governance/virtual-keys) for key creation, RBAC scoping, and budget caps.
GoRoute accepts either standard credential header:
- `Authorization: Bearer sk-gw-...` (OpenAI-compatible clients)
- `x-api-key: sk-gw-...` (Anthropic-compatible clients)
> [!NOTE]
> If a client sends **both** `Authorization: Bearer ...` and `x-api-key: ...` with different values, GoRoute rejects the request with `400 Bad Request` (`ambiguous_gateway_credentials`). Identical values in both headers are accepted.
---
## Integration Guides
### Coding Agents & IDEs
- [**Claude Code**](https://goroute-ai.github.io/docs/integrations/claude-code): Route Claude Code through `/anthropic` using `ANTHROPIC_BASE_URL` and `ANTHROPIC_AUTH_TOKEN` or `ANTHROPIC_API_KEY`.
- [**Cursor, Continue & Cline**](https://goroute-ai.github.io/docs/integrations/cursor-and-ides): Configure IDE extensions against `/v1` with Virtual Keys and unified model aliases.
- [**Codex CLI, Gemini CLI & OpenWebUI**](https://goroute-ai.github.io/docs/integrations/codex-and-cli-agents): Connect terminal coding agents and self-hosted chat UIs via `/v1`.
### Official Language SDKs & Agent Frameworks
- [**OpenAI SDKs (Python & TypeScript)**](https://goroute-ai.github.io/docs/integrations/openai-sdks): Chat completions, SSE streaming, embeddings, Responses API, and Realtime WebSockets.
- [**Anthropic SDKs (Python & TypeScript)**](https://goroute-ai.github.io/docs/integrations/anthropic-sdks): Native `/anthropic/v1/messages` and `/anthropic/v1/messages/count_tokens`.
- [**Agent Frameworks**](https://goroute-ai.github.io/docs/integrations/agent-frameworks): LangChain, OpenAI Agents SDK, and Google ADK with GoRoute MCP and A2A endpoints.
### Observability & Telemetry
- [**OpenTelemetry & Prometheus**](https://goroute-ai.github.io/docs/integrations/otel-and-prometheus): OTLP/HTTP span export (`tracing.otlp_endpoint`, `gen_ai.*` attributes, W3C `traceparent`) and Prometheus `/metrics` scraping.
---
## Claude Code
URL: https://goroute-ai.github.io/docs/integrations/claude-code
Markdown: https://goroute-ai.github.io/docs/integrations/claude-code.md
Section: Control Plane Documentation > Connect your tools > Coding tools and IDEs
# Claude Code
[Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) uses the Anthropic Messages API (`/v1/messages`) and token counting (`/v1/messages/count_tokens`). GoRoute exposes both endpoints under `/anthropic/v1/*`, so setting `ANTHROPIC_BASE_URL` to `/anthropic` routes all Claude Code traffic through your gateway with full Virtual Key budgets, RBAC, guardrails, and observability.
---
## 1. Prerequisites
1. A running GoRoute AI Gateway instance (for example, `http://localhost:8080`).
2. At least one Anthropic model (or alias such as `claude-sonnet-4-20250514` or `reasoning`) configured in `model_list` or via `/admin/v1/models`.
3. A Virtual Key (`sk-gw-...`) with permission to access the configured model(s).
---
## 2. Environment Configuration
Export `ANTHROPIC_BASE_URL` and your GoRoute Virtual Key before launching `claude`:
```bash
export ANTHROPIC_BASE_URL="http://localhost:8080/anthropic"
export ANTHROPIC_AUTH_TOKEN="sk-gw-your-virtual-key"
```
Alternatively, if you use `ANTHROPIC_API_KEY` (which Claude Code sends via `x-api-key`), set:
```bash
export ANTHROPIC_BASE_URL="http://localhost:8080/anthropic"
export ANTHROPIC_API_KEY="sk-gw-your-virtual-key"
```
> [!WARNING]
> Do not set **both** `ANTHROPIC_AUTH_TOKEN` and `ANTHROPIC_API_KEY` to different values in the same shell. GoRoute validates both `Authorization: Bearer` and `x-api-key` and returns `400 ambiguous_gateway_credentials` if they conflict.
Launch Claude Code:
```bash
claude
```
---
## 3. Model Mapping & Aliases
If your organization uses GoRoute unified aliases (e.g., `fast`, `reasoning`) or routes Claude Code requests to non-Anthropic backends via GoRoute's Anthropic-to-canonical translator, pass `--model` or set `ANTHROPIC_MODEL`:
```bash
export ANTHROPIC_MODEL="claude-sonnet-4-20250514"
claude --model reasoning
```
GoRoute's `/anthropic/v1/messages` handler normalizes system blocks, `extended_thinking` budgets, and parallel `tool_result` turns automatically. See [Anthropic Messages Ingress API](https://goroute-ai.github.io/docs/gateway/endpoints/messages) for translation details.
---
## 4. Connecting Claude Code to GoRoute MCP
You can also point Claude Code at GoRoute's [MCP Virtual Hub](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest) so MCP tool calls inherit your Virtual Key RBAC and sandboxing:
```json
{
"mcpServers": {
"goroute-hub": {
"type": "http",
"url": "http://localhost:8080/mcp",
"headers": {
"Authorization": "Bearer sk-gw-your-virtual-key"
}
}
}
}
```
---
## 5. Verification
Confirm the `/anthropic/v1/messages` route responds with your Virtual Key before starting a session:
```bash
curl -sS http://localhost:8080/anthropic/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: sk-gw-your-virtual-key" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "claude-sonnet-4-20250514",
"max_tokens": 64,
"messages": [{"role": "user", "content": "Say hello in one word."}]
}'
```
For repository-level agent workflow patterns (`CLAUDE.md`, `.claude/rules/`, and `AGENTS.md`), see [Engineering Guide for Coding Agents](https://goroute-ai.github.io/docs/contributing/claude-code-and-agents).
---
## Cursor, Continue & Cline
URL: https://goroute-ai.github.io/docs/integrations/cursor-and-ides
Markdown: https://goroute-ai.github.io/docs/integrations/cursor-and-ides.md
Section: Control Plane Documentation > Connect your tools > Coding tools and IDEs
# Cursor, Continue & Cline
AI-native editors and IDE extensions connect to GoRoute AI Gateway through its OpenAI-compatible `/v1` data plane (`POST /v1/chat/completions`, `GET /v1/models`). Centralizing IDE traffic through GoRoute gives engineering teams unified model aliases, per-developer Virtual Keys, spend caps, and zero-leakage log scrubbing.
---
## Cursor
Cursor supports custom OpenAI-compatible base URLs in **Settings → Models**:
1. Open **Cursor Settings** (`Cmd+,` / `Ctrl+,`) → **Models**.
2. Expand **OpenAI API Key** and enable **Override OpenAI Base URL**.
3. Set the base URL to your GoRoute `/v1` endpoint:
```text
https://gateway.example.com/v1
```
*(For local development without TLS, use `http://localhost:8080/v1`; note that Cursor's cloud-proxied features require a publicly reachable HTTPS endpoint such as a tunnel or ingress.)*
4. Paste your GoRoute Virtual Key (`sk-gw-...`) into **OpenAI API Key** and click **Verify**.
5. Add the model names or aliases configured in your GoRoute `model_list` (for example `gpt-4o`, `claude-sonnet-4-20250514`, or `reasoning`).
---
## Continue (`config.yaml` / `config.json`)
[Continue](https://docs.continue.dev/) supports GoRoute directly via its `openai` provider with `apiBase`.
```yaml title="~/.continue/config.yaml"
name: GoRoute Workspace
version: 1.0.0
schema: v1
models:
- name: GoRoute Reasoning
provider: openai
model: reasoning
apiBase: http://localhost:8080/v1
apiKey: sk-gw-your-virtual-key
- name: GoRoute Fast Autocomplete
provider: openai
model: fast
apiBase: http://localhost:8080/v1
apiKey: sk-gw-your-virtual-key
roles:
- autocomplete
```
---
## Cline (VS Code / Cursor Extension)
[Cline](https://github.com/cline/cline) supports both OpenAI-compatible and Anthropic-compatible providers:
### Option A: OpenAI Compatible (Recommended for Multi-Provider Aliases)
1. Open Cline **Settings** (gear icon).
2. Set **API Provider** to **OpenAI Compatible**.
3. Set **Base URL** to `http://localhost:8080/v1`.
4. Set **API Key** to your Virtual Key (`sk-gw-...`).
5. Set **Model ID** to any model or alias configured in GoRoute (e.g., `claude-sonnet-4-20250514` or `gpt-4o`).
### Option B: Anthropic Native Mode
1. Set **API Provider** to **Anthropic**.
2. Enable **Use custom base URL** and set it to `http://localhost:8080/anthropic`.
3. Set **Anthropic API Key** to your Virtual Key (`sk-gw-...`).
---
## Verifying IDE Connectivity
Confirm your Virtual Key can list models and complete a chat request against `/v1`:
```bash
curl -sS http://localhost:8080/v1/models \
-H "Authorization: Bearer sk-gw-your-virtual-key"
```
---
## Codex CLI, Gemini CLI & OpenWebUI
URL: https://goroute-ai.github.io/docs/integrations/codex-and-cli-agents
Markdown: https://goroute-ai.github.io/docs/integrations/codex-and-cli-agents.md
Section: Control Plane Documentation > Connect your tools > Coding tools and IDEs
# Codex CLI, Gemini CLI & OpenWebUI
Terminal coding agents and self-hosted chat frontends can point directly at GoRoute's `/v1` endpoints using standard OpenAI-compatible environment variables and Virtual Keys (`sk-gw-...`).
---
## OpenAI Codex CLI
[Codex CLI](https://github.com/openai/codex) uses the OpenAI Responses API (`/v1/responses`) and Chat Completions (`/v1/chat/completions`), both of which GoRoute serves under `/v1` and `/openai/v1`.
### Environment Variables
```bash
export OPENAI_BASE_URL="http://localhost:8080/v1"
export OPENAI_API_KEY="sk-gw-your-virtual-key"
codex --model gpt-4.1
```
### TOML Configuration (`~/.codex/config.toml`)
```toml title="~/.codex/config.toml"
model = "gpt-4.1"
model_provider = "goroute"
[model_providers.goroute]
name = "GoRoute AI Gateway"
base_url = "http://localhost:8080/v1"
env_key = "GOROUTE_API_KEY"
wire_api = "responses"
```
Then export your Virtual Key and run `codex`:
```bash
export GOROUTE_API_KEY="sk-gw-your-virtual-key"
codex
```
---
## Gemini CLI
When using Gemini models configured in GoRoute (such as `gemini/gemini-2.5-flash`), you can route OpenAI-compatible CLI workflows through `/v1`:
```bash
export OPENAI_BASE_URL="http://localhost:8080/v1"
export OPENAI_API_KEY="sk-gw-your-virtual-key"
```
If you are calling GoRoute directly from scripts or tools expecting a Gemini model alias, configure the model in `config.yaml`:
```yaml
model_list:
- model_name: gemini-2.5-flash
model_params:
model: gemini/gemini-2.5-flash
api_key: os.environ/GEMINI_API_KEY
```
GoRoute's Gemini adapter translates `/v1/chat/completions` and `/v1/embeddings` requests into native Gemini API calls while enforcing Virtual Key budgets and SSRF egress policies.
---
## OpenWebUI
[OpenWebUI](https://docs.openwebui.com/) connects to GoRoute as an OpenAI-compatible backend, automatically discovering available models via `GET /v1/models`.
### Docker Environment Variables
```bash
docker run -d -p 3000:8080 \
-e OPENAI_API_BASE_URL="http://host.docker.internal:8080/v1" \
-e OPENAI_API_KEY="sk-gw-your-virtual-key" \
--name open-webui \
ghcr.io/open-webui/open-webui:main
```
### Admin Panel Configuration
1. Sign in to OpenWebUI as an administrator and go to **Admin Panel → Settings → Connections**.
2. Under **OpenAI API**, set:
- **URL**: `http://localhost:8080/v1` (or your cluster service URL, e.g., `http://ai-gateway:8080/v1`)
- **API Key**: `sk-gw-your-virtual-key`
3. Click **Verify Connection** to load your GoRoute model catalog.
---
## OpenAI SDKs (Python & TypeScript)
URL: https://goroute-ai.github.io/docs/integrations/openai-sdks
Markdown: https://goroute-ai.github.io/docs/integrations/openai-sdks.md
Section: Control Plane Documentation > Connect your tools > SDKs and agent frameworks
# OpenAI SDKs (Python & TypeScript)
The official `openai` SDKs for Python and TypeScript work with GoRoute AI Gateway by overriding `base_url` / `baseURL` to `http://localhost:8080/v1` and passing a GoRoute Virtual Key (`sk-gw-...`) as `api_key` / `apiKey`.
---
## 1. Client Initialization
#### Python (openai)
```python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="sk-gw-your-virtual-key",
)
```
#### TypeScript (openai)
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:8080/v1",
apiKey: "sk-gw-your-virtual-key",
});
```
---
## 2. Chat Completions (`POST /v1/chat/completions`)
#### Python
```python
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "Explain circuit breakers in one sentence."},
],
)
print(response.choices[0].message.content)
```
#### TypeScript
```typescript
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "Explain circuit breakers in one sentence." },
],
});
console.log(response.choices[0].message.content);
```
---
## 3. Streaming (`stream: true`)
#### Python
```python
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Count from 1 to 5."}],
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
```
#### TypeScript
```typescript
const stream = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [{ role: "user", content: "Count from 1 to 5." }],
stream: true,
});
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) process.stdout.write(delta);
}
```
---
## 4. Embeddings (`POST /v1/embeddings`)
#### Python
```python
emb = client.embeddings.create(
model="text-embedding-3-small",
input="GoRoute AI Gateway",
)
print(len(emb.data[0].embedding))
```
#### TypeScript
```typescript
const emb = await client.embeddings.create({
model: "text-embedding-3-small",
input: "GoRoute AI Gateway",
});
console.log(emb.data[0].embedding.length);
```
---
## 5. Responses API (`POST /v1/responses` & `/openai/v1/responses`)
GoRoute implements the stateful/agentic Responses API at both `/v1/responses` and `/openai/v1/responses`:
#### Python
```python
resp = client.responses.create(
model="gpt-4.1-mini",
input="Summarize the benefits of micro-dollar accounting.",
)
print(resp.output_text)
```
#### TypeScript
```typescript
const resp = await client.responses.create({
model: "gpt-4.1-mini",
input: "Summarize the benefits of micro-dollar accounting.",
});
console.log(resp.output_text);
```
---
## 6. Realtime WebSocket (`GET /v1/realtime` & `/openai/v1/realtime`)
For bidirectional audio/text streaming over WebSockets, connect to `ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview` (or `wss://...` in production) with `Authorization: Bearer sk-gw-your-virtual-key`:
```typescript
import WebSocket from "ws";
const ws = new WebSocket(
"ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview",
{
headers: {
Authorization: "Bearer sk-gw-your-virtual-key",
},
}
);
ws.on("open", () => {
ws.send(JSON.stringify({ type: "response.create" }));
});
```
See [Realtime WebSocket API](https://goroute-ai.github.io/docs/gateway/endpoints/realtime) and [Gateway API Reference](https://goroute-ai.github.io/docs/api/gateway-api) for the full list of supported OpenAI-compatible endpoints.
---
## Anthropic SDKs (Python & TypeScript)
URL: https://goroute-ai.github.io/docs/integrations/anthropic-sdks
Markdown: https://goroute-ai.github.io/docs/integrations/anthropic-sdks.md
Section: Control Plane Documentation > Connect your tools > SDKs and agent frameworks
# Anthropic SDKs (Python & TypeScript)
The official Anthropic SDKs (`anthropic` in Python and `@anthropic-ai/sdk` in TypeScript) append `/v1/messages` and `/v1/messages/count_tokens` to their configured base URL. Pointing `base_url` / `baseURL` to `http://localhost:8080/anthropic` routes requests directly to GoRoute's `/anthropic/v1/messages` and `/anthropic/v1/messages/count_tokens` endpoints.
---
## 1. Client Initialization
Pass your GoRoute Virtual Key (`sk-gw-...`) as `api_key` / `apiKey` (sent via `x-api-key`) or `auth_token` / `authToken` (sent via `Authorization: Bearer`).
#### Python (anthropic)
```python
from anthropic import Anthropic
client = Anthropic(
base_url="http://localhost:8080/anthropic",
api_key="sk-gw-your-virtual-key",
)
```
#### TypeScript (@anthropic-ai/sdk)
```typescript
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "http://localhost:8080/anthropic",
apiKey: "sk-gw-your-virtual-key",
});
```
> [!NOTE]
> Avoid setting both `api_key` (`x-api-key`) and `auth_token` (`Authorization: Bearer`) to different values. GoRoute rejects conflicting credential headers with `400 ambiguous_gateway_credentials`.
---
## 2. Messages (`POST /anthropic/v1/messages`)
#### Python
```python
message = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=256,
system="You are a concise systems engineer.",
messages=[
{"role": "user", "content": "What is a sliding-window rate limiter?"}
],
)
print(message.content[0].text)
```
#### TypeScript
```typescript
const message = await client.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 256,
system: "You are a concise systems engineer.",
messages: [
{ role: "user", content: "What is a sliding-window rate limiter?" },
],
});
console.log(message.content[0]);
```
---
## 3. Streaming Messages (`stream: true`)
#### Python
```python
with client.messages.stream(
model="claude-sonnet-4-20250514",
max_tokens=256,
messages=[{"role": "user", "content": "List three Go concurrency primitives."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
#### TypeScript
```typescript
const stream = client.messages.stream({
model: "claude-sonnet-4-20250514",
max_tokens: 256,
messages: [{ role: "user", content: "List three Go concurrency primitives." }],
});
stream.on("text", (text) => process.stdout.write(text));
await stream.finalMessage();
```
---
## 4. Token Counting (`POST /anthropic/v1/messages/count_tokens`)
GoRoute computes pre-flight token counts locally without forwarding to the upstream provider:
#### Python
```python
count = client.messages.count_tokens(
model="claude-sonnet-4-20250514",
messages=[{"role": "user", "content": "Estimate tokens for this prompt."}],
)
print(count.input_tokens)
```
#### TypeScript
```typescript
const count = await client.messages.countTokens({
model: "claude-sonnet-4-20250514",
messages: [{ role: "user", content: "Estimate tokens for this prompt." }],
});
console.log(count.input_tokens);
```
See [Anthropic Messages Ingress API](https://goroute-ai.github.io/docs/gateway/endpoints/messages) for details on extended thinking blocks, tool schema translation, and alternating turn consolidation.
---
## Agent Frameworks (LangChain, OpenAI Agents SDK & Google ADK)
URL: https://goroute-ai.github.io/docs/integrations/agent-frameworks
Markdown: https://goroute-ai.github.io/docs/integrations/agent-frameworks.md
Section: Control Plane Documentation > Connect your tools > SDKs and agent frameworks
# Agent Frameworks (LangChain, OpenAI Agents SDK & Google ADK)
Agentic frameworks can route both **LLM inference** (`/v1` or `/anthropic`) and **tool/agent protocols** ([MCP](https://goroute-ai.github.io/docs/agents/mcp-overview) and [A2A](https://goroute-ai.github.io/docs/agents/a2a-overview)) through GoRoute AI Gateway.
---
## 1. LangChain (Python & TypeScript)
Point `ChatOpenAI` or `ChatAnthropic` at GoRoute using your Virtual Key (`sk-gw-...`):
#### Python (langchain-openai)
```python
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="reasoning",
base_url="http://localhost:8080/v1",
api_key="sk-gw-your-virtual-key",
)
resp = llm.invoke("Design a fallback chain for 503 outages.")
print(resp.content)
```
#### TypeScript (@langchain/openai)
```typescript
import { ChatOpenAI } from "@langchain/openai";
const llm = new ChatOpenAI({
model: "reasoning",
apiKey: "sk-gw-your-virtual-key",
configuration: {
baseURL: "http://localhost:8080/v1",
},
});
const resp = await llm.invoke("Design a fallback chain for 503 outages.");
console.log(resp.content);
```
---
## 2. OpenAI Agents SDK
The [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) works with GoRoute by passing a custom `AsyncOpenAI` client configured with `base_url="http://localhost:8080/v1"` and your Virtual Key:
```python
from openai import AsyncOpenAI
from agents import Agent, Runner, set_default_openai_client
client = AsyncOpenAI(
base_url="http://localhost:8080/v1",
api_key="sk-gw-your-virtual-key",
)
set_default_openai_client(client)
agent = Agent(
name="TriageAgent",
model="gpt-4.1-mini",
instructions="Route infrastructure questions concisely.",
)
result = await Runner.run(agent, "How does GoRoute enforce SSRF protection?")
print(result.final_output)
```
---
## 3. Google ADK (Agent Development Kit)
In Google ADK, you can either route model calls through GoRoute's OpenAI-compatible `/v1` endpoint or expose/consume remote agents over GoRoute's **A2A v1.0** plane:
1. **Model Routing via `/v1`**:
Set `OPENAI_BASE_URL="http://localhost:8080/v1"` and `OPENAI_API_KEY="sk-gw-your-virtual-key"` for OpenAI-compatible model wrappers, with Gemini models declared in GoRoute's `model_list` (e.g., `gemini/gemini-2.5-flash`).
2. **A2A Agent Discovery & Dispatch**:
GoRoute serves A2A agent cards at `GET /.well-known/agent-card.json` and task endpoints under `/a2a/v1/agents/{agentSlug}/` (`POST /`, `GET /tasks`, `GET /tasks/{taskID}`, `POST /tasks/{taskID}:cancel`, `GET|POST /tasks/{taskID}:subscribe`). See [A2A Agent Cards](https://goroute-ai.github.io/docs/agents/a2a-agent-cards).
---
## 4. Connecting Frameworks to GoRoute MCP
Any framework supporting **Streamable HTTP MCP** or **REST tool calls** can consume tools governed by GoRoute's MCP Virtual Hub:
| Protocol | Endpoint | Auth Header | Guide |
| :--- | :--- | :--- | :--- |
| **MCP Virtual Hub (All Allowed Tools)** | `POST http://localhost:8080/mcp` | `Authorization: Bearer sk-gw-...` | [MCP Hub & REST](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest) |
| **Scoped MCP Server** | `POST http://localhost:8080/mcp/{serverSlug}` | `Authorization: Bearer sk-gw-...` | [MCP Overview](https://goroute-ai.github.io/docs/agents/mcp-overview) |
| **Curated MCP Toolset** | `POST http://localhost:8080/mcp/toolsets/{toolsetSlug}` | `Authorization: Bearer sk-gw-...` | [Progressive Discovery](https://goroute-ai.github.io/docs/agents/mcp-progressive-discovery) |
| **MCP REST Bridge** | `GET /mcp-rest/tools/list`, `POST /mcp-rest/tools/call` | `Authorization: Bearer sk-gw-...` | [MCP Hub & REST](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest) |
---
## OpenTelemetry & Prometheus
URL: https://goroute-ai.github.io/docs/integrations/otel-and-prometheus
Markdown: https://goroute-ai.github.io/docs/integrations/otel-and-prometheus.md
Section: Control Plane Documentation > Connect your tools > Monitoring and telemetry
# OpenTelemetry & Prometheus
GoRoute AI Gateway provides built-in **OTLP/HTTP distributed tracing** (`internal/observability/tracing_otlp.go`) and **Prometheus 0.0.4 metrics** (`GET /metrics`). All outbound OTLP exports pass through GoRoute's outbound SSRF `network.Policy`.
> [!NOTE]
> This integration guide covers connecting external collectors (**OTel Collector**, **Jaeger**, **Grafana Tempo**, and **Prometheus** scrapers). For built-in GoRoute telemetry (`gateway_*` metric definitions, W3C `traceparent` generation, `/healthz` / `/readyz` probes, and secret-scrubbed logging), see [Observability, Metrics & Health Probes](https://goroute-ai.github.io/docs/deploy/observability).
---
## 1. OpenTelemetry (OTLP/HTTP) Configuration
Configure `tracing` in `config.yaml`:
```yaml title="config.yaml"
tracing:
otlp_endpoint: "https://otel-collector.monitoring.svc.cluster.local:4318/v1/traces"
service_name: "ai-gateway-go"
headers:
Authorization: "Bearer ${OTEL_EXPORTER_TOKEN}"
```
### Configuration Fields
| Field | Default | Behavior |
| :--- | :--- | :--- |
| `tracing.otlp_endpoint` | `""` (disabled / no-op tracer) | OTLP/HTTP collector URL. If the path is empty or `/`, GoRoute automatically appends `/v1/traces`. Validated against the outbound `network.Policy` at startup. |
| `tracing.service_name` | `"ai-gateway-go"` | Sets the OpenTelemetry `service.name` resource attribute. |
| `tracing.headers` | `{}` | Optional key-value headers attached to OTLP/HTTP export requests (e.g., `Authorization`, `x-scope-orgid`). |
> [!IMPORTANT]
> Because `tracing.otlp_endpoint` is validated by `internal/security/network`, loopback or private collector addresses (such as `http://localhost:4318` or cluster-internal services) require the collector host/CIDR to be permitted in your `network` security policy if private-IP blocking is enabled.
---
## 2. W3C `traceparent` & `gen_ai.*` Span Attributes
GoRoute's `TraceMiddleware` extracts incoming W3C `traceparent` (or `X-Trace-Id`) headers, generates a 128-bit TraceID + 64-bit SpanID when absent, echoes `traceparent` and `X-Trace-Id` on responses, and propagates `traceparent` to upstream providers.
Upstream inference spans are emitted as `gen_ai.client` with OpenTelemetry GenAI semantic conventions:
| Attribute | Description |
| :--- | :--- |
| `gen_ai.operation.name` | Operation name (e.g., `chat`, `embeddings`). |
| `gen_ai.system` | Upstream provider identifier (e.g., `openai`, `anthropic`, `gemini`). |
| `gen_ai.request.model` | Target model name dispatched to the provider. |
| `gen_ai.usage.input_tokens` | Prompt / input token count. |
| `gen_ai.usage.output_tokens` | Completion / output token count. |
| `gen_ai.usage.total_tokens` | Total tokens consumed. |
---
## 3. OTel Collector, Jaeger & Grafana Tempo
Point `tracing.otlp_endpoint` to your OpenTelemetry Collector HTTP receiver (port `4318`), then fan out to Jaeger or Grafana Tempo:
```yaml title="otel-collector-config.yaml"
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
exporters:
otlp/tempo:
endpoint: tempo.monitoring.svc.cluster.local:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlp/tempo]
```
- **Jaeger (v1.35+ native OTLP)**: Point `tracing.otlp_endpoint` directly to `http://:4318/v1/traces`.
- **Grafana Tempo**: Point `tracing.otlp_endpoint` to `http://:4318/v1/traces` (or route via OTel Collector above).
---
## 4. Prometheus Scraping (`GET /metrics`)
GoRoute exposes Prometheus text exposition format `0.0.4` at `GET /metrics`. The endpoint is protected by `metricsAuthMiddleware` and requires the **Master Admin Key** in `Authorization: Bearer ` or `x-api-key: `.
```yaml title="deployments/prometheus/prometheus.yml"
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "ai-gateway-cluster"
authorization:
type: Bearer
credentials_file: /etc/prometheus/secrets/master_admin_key
static_configs:
- targets: ["ai-gateway-1:8080", "ai-gateway-2:8080"]
labels:
cluster: "prod-verification"
```
For the full catalog of exported `gateway_*` metrics, health probes (`/healthz`, `/readyz`), and structured log scrubbing, see [Observability, Metrics & Health Probes](https://goroute-ai.github.io/docs/deploy/observability).
---
## Complete Configuration and Environment Reference
URL: https://goroute-ai.github.io/docs/deploy/configuration
Markdown: https://goroute-ai.github.io/docs/deploy/configuration.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Complete Configuration and Environment Reference
GoRoute AI Gateway supports a layered configuration architecture:
1. **Built-in Defaults**: Hardcoded, zero-dependency defaults optimized for instant developer startup.
2. **Declarative YAML (`configs/config.yaml`)**: Structured configuration defining storage backends, security boundaries, rate limits, model groups, and routing topologies.
3. **Environment Variable Overrides (`GATEWAY_*` / `os.environ/VAR`)**: Production overrides for Kubernetes ConfigMaps, container runtime secrets, and 12-factor cloud deployments.
4. **Go SDK Options (`pkg/gateway`)**: In-process configuration primitives for embedded architectures.
---
## 1. Quickstart Configuration Modes
### Mode A: Zero-Config Environment Variables (Fastest)
Set your upstream provider API keys directly in your shell or container definition. The gateway automatically spins up an in-memory repository, provisions an AES-256-GCM vault, and configures default model aliases:
```bash
# Frontier LLM Providers
export OPENAI_API_KEY="sk-proj-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="AIzaSy..."
export MISTRAL_API_KEY="mis_..."
# Enterprise Cloud Hyperscalers
export AWS_ACCESS_KEY_ID="AKIA..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_REGION="us-east-1"
export GCP_PROJECT_ID="my-gcp-project"
export GCP_REGION="us-central1"
# Gateway Master Secrets
export GATEWAY_MASTER_ADMIN_KEY="sk-gw-master-secret-passphrase-change-me"
export GATEWAY_MASTER_ENCRYPTION_KEY="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
# Launch Gateway
./ai-gateway
```
### Mode B: Production Declarative YAML
Supply the path to your YAML configuration file via the `-config` CLI argument:
```bash
./ai-gateway -config /etc/goroute/config.yaml
```
The gateway resolves environment variable secret placeholders matching `os.environ/VARIABLE_NAME` inside strings dynamically during YAML parsing:
```yaml
model_params:
model: "openai/gpt-4o"
api_key: "os.environ/OPENAI_API_KEY"
```
---
## 2. Complete 70+ Configuration Options Matrix
The table below documents every configurable parameter across all gateway subsystems.
| YAML Path | Environment Variable Override | Default | Subsystem & Description |
| :--- | :--- | :--- | :--- |
| `server.host` | None | `"0.0.0.0"` | **Server**: HTTP network bind address. |
| `server.environment` | `ENV` / `GATEWAY_ENV` | `"development"` | **Server**: Runtime environment (`"development"`, `"staging"`, `"production"`). Production rejects default secrets. |
| `server.port` | `GATEWAY_PORT` | `8080` | **Server**: HTTP listen port (`1`–`65535`). |
| `server.grpc_port` | `GATEWAY_GRPC_PORT` | `0` | **Server**: Optional A2A gRPC listen port (`0` disables gRPC). |
| `server.cors_allowed_origins` | None | `[]` | **Security**: Allowed CORS origins (empty disables CORS headers). |
| `server.cors_allow_credentials` | None | `false` | **Security**: Enables `Access-Control-Allow-Credentials` (disallowed with `*`). |
| `server.enable_hsts` | None | `false` | **Security**: Emits `Strict-Transport-Security` header on responses. |
| `server.max_file_storage_bytes_per_org` | `GATEWAY_MAX_FILE_STORAGE_BYTES_PER_ORG` | `5368709120` | **Files**: Per-org storage quota in bytes (`5 GiB` default; `0` = unlimited). |
| `server.max_concurrent_file_uploads` | `GATEWAY_MAX_CONCURRENT_FILE_UPLOADS` | `8` | **Files**: Maximum concurrent `/v1/files` uploads (`0` = unlimited). |
| `server.read_timeout` | None | `30s` | **Server**: Maximum duration for reading the entire request body. |
| `server.write_timeout` | None | `300s` | **Server**: Maximum duration before timing out writes (sized for reasoning streams). |
| `server.idle_timeout` | None | `120s` | **Server**: Maximum time to wait for the next request when keep-alives are enabled. |
| `server.shutdown_timeout` | None | `15s` | **Server**: Graceful shutdown deadline before force-closing active HTTP streams. |
| `logging.level` | `GATEWAY_LOG_LEVEL` | `"info"` | **Logging**: Log verbosity: `"debug"`, `"info"`, `"warn"`, `"error"`. |
| `logging.format` | `GATEWAY_LOG_FORMAT` | `"text"` | **Logging**: `"json"` (cloud structured) or `"text"` (CLI). Auto-selects `"json"` if `ENV=production`. |
| `logging.add_source` | `GATEWAY_LOG_ADD_SOURCE` | `false` | **Logging**: Annotates log lines with source `file:line` for developer debugging. |
| `storage.mode` | `GATEWAY_STORAGE_MODE` | `"memory"` | **Storage**: Repository backend: `"memory"` or `"postgres"`. |
| `storage.postgres_dsn` | `GATEWAY_POSTGRES_DSN` | `""` | **Storage**: PostgreSQL connection DSN (`postgres://user:pass@host:5432/db`). |
| `storage.redis_addr` | `GATEWAY_REDIS_ADDR` | `"localhost:6379"` | **Storage**: Standalone Redis host:port for L2 auth caching, Lua rate limits, and pub/sub. |
| `storage.redis_addrs` | `GATEWAY_REDIS_ADDRS` | `[]` | **Storage**: Comma-separated Redis Cluster or Sentinel seed nodes. |
| `storage.redis_master_name` | `GATEWAY_REDIS_MASTER_NAME` | `""` | **Storage**: Redis Sentinel master name. |
| `tracing.otlp_endpoint` | None | `""` | **Observability**: Optional OTLP/HTTP traces URL (empty disables export). |
| `tracing.service_name` | None | `"ai-gateway-go"` | **Observability**: Service name attribute attached to exported OpenTelemetry spans. |
| `tracing.headers` | None | `{}` | **Observability**: Optional HTTP headers sent with OTLP trace exports (e.g., auth tokens). |
| `storage.redis_pass` | `GATEWAY_REDIS_PASS` | `""` | **Storage**: Optional Redis authentication password. |
| `storage.redis_db` | None | `0` | **Storage**: Redis logical database index (`0`–`15`). |
| `auth.master_admin_key` | `GATEWAY_MASTER_ADMIN_KEY` | *(auto-generated)* | **Auth**: Master Bearer token for `/admin/v1/*` control plane APIs. |
| `auth.allow_master_key_data_plane` | `GATEWAY_ALLOW_MASTER_KEY_DATA_PLANE` | `false` | **Auth**: Allows master admin key on data-plane routes (disabled by default for security). |
| `auth.master_encryption_key` | `GATEWAY_MASTER_ENCRYPTION_KEY` | *(auto-generated)* | **Auth**: 64-character hex string (32-byte key) for AES-256-GCM vault encryption. |
| `auth.jwt_secret` | `GATEWAY_JWT_SECRET` | *(random)* | **Auth**: HMAC secret used for signing internal short-lived JWT tokens. |
| `auth.default_org_id` | None | `"org_default"` | **Auth**: Primary root organization identifier seeded on initial startup. |
| `auth.default_virtual_key` | `GATEWAY_DEFAULT_KEY` | `""` | **Auth**: Optional pre-seeded client virtual key for automated integration testing. |
| `auth.l1_cache_ttl` | None | `15s` | **Auth**: In-memory L1 cache TTL for virtual key validation and identity contexts. |
| `auth.l2_cache_ttl` | None | `5m` | **Auth**: Redis L2 cache TTL for virtual key validation across clustered pods. |
| `router.default_fast_alias` | None | `"fast"` | **Router**: Virtual model alias routing to high-throughput, low-latency models. |
| `router.default_reasoning_alias` | None | `"reasoning"` | **Router**: Virtual model alias routing to frontier deep-reasoning models. |
| `router.default_coding_alias` | None | `"coding"` | **Router**: Virtual model alias routing to specialized code-generation models. |
| `router.complexity_threshold` | None | `0.40` | **Router**: Prompt complexity heuristic score `[0.0, 1.0]` triggering `"auto"` $\rightarrow$ `"reasoning"`. |
| `router.long_context_tokens` | None | `32000` | **Router**: Input token threshold triggering automatic routing to long-context engines. |
| `router.default_max_retries` | None | `2` | **Router**: Maximum intra-deployment retries before failing over to fallback chain. |
| `router.base_retry_delay` | `GATEWAY_ROUTER_BASE_RETRY_DELAY` | `50ms` | **Router**: Base backoff delay for full-jitter exponential retry calculations. |
| `router.max_retry_delay` | `GATEWAY_ROUTER_MAX_RETRY_DELAY` | `2s` | **Router**: Maximum ceiling delay between retry attempts. |
| `router.default_cooldown` | `GATEWAY_ROUTER_DEFAULT_COOLDOWN` | `60s` | **Router**: Duration a tripped deployment remains isolated in `OPEN` circuit breaker state. |
| `router.failure_window` | None | `30s` | **Router**: Sliding time window for tracking consecutive upstream provider errors. |
| `router.failure_threshold` | None | `3` | **Router**: Number of consecutive errors required to trip deployment circuit breaker. |
| `rate_limit.enabled` | None | `true` | **Rate Limit**: Enables distributed sliding-window rate limiting via Redis Lua scripts. |
| `rate_limit.window_seconds` | None | `60` | **Rate Limit**: Window duration in seconds for RPM and TPM rate tracking. |
| `rate_limit.concurrency_ttl_sec` | None | `300` | **Rate Limit**: TTL lease in seconds for tracking concurrent in-flight requests. |
| `rate_limit.default_estimated_tpm` | None | `128` | **Rate Limit**: Heuristic token estimate reserved prior to parsing request payload. |
| `rate_limit.redis_timeout` | None | `50ms` | **Rate Limit**: Timeout for Redis rate check execution before failing closed. |
| `finops.channel_buffer_size` | None | `16384` | **FinOps**: Asynchronous ring buffer capacity for in-flight spend ledger events. |
| `finops.batch_size` | None | `250` | **FinOps**: Maximum spend records batched into a single database bulk insert. |
| `finops.flush_interval` | `GATEWAY_FINOPS_FLUSH_INTERVAL` | `500ms` | **FinOps**: Maximum interval between background spend ledger flushes. |
| `finops.max_unflushed_age` | None | `5s` | **FinOps**: Hard timeout forcing asynchronous spend ledger buffer flush. |
| `discovery.auto_approve_discovered_models` | None | `true` | **Discovery**: Automatically activates newly discovered upstream models. |
| `discovery.poll_interval_hours` | None | `24` | **Discovery**: Interval in hours between background upstream provider catalog syncs. |
| `discovery.enable_background_sync` | None | `true` | **Discovery**: Enables background discovery daemon across enabled providers. |
| `security.network.egress_proxy_url` | `GATEWAY_EGRESS_PROXY_URL` | `""` | **Network**: Corporate HTTP/HTTPS egress proxy URL for enterprise firewalls. |
| `security.network.outbound_ssrf.allow_private_networks` | `GATEWAY_SSRF_ALLOW_PRIVATE` | `false` | **SSRF**: Enables egress to private/internal IPs (must be `false` in production). |
| `security.network.outbound_ssrf.allowed_private_cidrs` | None | `[]` | **SSRF**: Explicit CIDR allowlist (e.g. `["10.240.0.0/16"]`) when egress is gated. |
| `security.network.outbound_ssrf.allow_insecure_http` | `GATEWAY_SSRF_ALLOW_INSECURE_HTTP` | `false` | **SSRF**: Permits unencrypted HTTP upstream calls (prohibited in production). |
| `security.network.outbound_ssrf.allowed_schemes` | None | `["https"]` | **SSRF**: Permitted URL schemes for provider endpoints (`https`, `http`). |
| `security.network.outbound_ssrf.blocked_ips` | None | `["169.254.169.254", ...]` | **SSRF**: Cloud instance metadata IPs blocked from outbound routing. |
| `security.network.outbound_ssrf.dial_timeout` | None | `10s` | **SSRF**: Network connect and TLS handshake timeout for outbound calls. |
| `security.network.outbound_ssrf.max_redirects` | None | `3` | **SSRF**: Maximum allowed HTTP redirect hops on outbound provider requests. |
| `security.network.inbound_caller.expose_on_public_internet` | None | `false` | **Network**: Policy flag asserting whether gateway is exposed to public web. |
| `security.network.inbound_caller.allowed_client_cidrs` | None | `[]` | **Network**: Client CIDR allowlist restricting incoming caller IP addresses. |
| `mcp.enabled` | None | `true` | **MCP**: Activates Model Context Protocol virtual hub and bridge endpoints. |
| `mcp.allow_stdio_servers` | None | `false` | **MCP**: Permits spawning local command-line subprocess MCP tool servers. |
| `mcp.session_ttl_minutes` | None | `15` | **MCP**: Inactivity timeout for stateful 2025-11-25 MCP SSE sessions. |
| `mcp.execution_timeout_seconds` | None | `30` | **MCP**: Maximum execution duration for individual MCP tool calls. |
| `mcp.max_payload_bytes` | None | `10485760` | **MCP**: Maximum allowed tool request and response payload size (10 MB). |
| `mcp.handle_ttl` | None | `1h` | **MCP**: Cache TTL for 256-bit progressive discovery tool handles (`tool_hdl_...`). |
| `a2a.enabled` | None | `true` | **A2A**: Activates Linux Foundation Agent-to-Agent v1.0 protocol router. |
| `a2a.allow_http_binding` | None | `true` | **A2A**: Enables standard REST HTTP+JSON binding on `/a2a/v1/*`. |
| `a2a.allow_jsonrpc_binding` | None | `true` | **A2A**: Enables JSON-RPC 2.0 protocol binding on `/a2a/v1/*`. |
| `a2a.allow_legacy_card_discovery` | None | `false` | **A2A**: Allows unsigned or legacy uncanonicalized agent card discovery. |
| `a2a.max_task_lifespan` | None | `24h` | **A2A**: Retention period for active and completed A2A task records. |
| `a2a.stream_buffer_capacity` | None | `256` | **A2A**: Ring buffer event capacity for SSE task streaming subscribers. |
| `a2a.stream_publish_interval` | `GATEWAY_A2A_STREAM_PUBLISH_INTERVAL` | `50ms` | **A2A**: Polling interval for background task stream publication daemon. |
| `a2a.max_push_retries` | `GATEWAY_A2A_MAX_PUSH_RETRIES` | `5` | **A2A**: Maximum webhook push notification delivery retries before Dead Lettering. |
| `a2a.signing_key_id` | `GATEWAY_A2A_SIGNING_KEY_ID` | `"gw-a2a-signer-v1"` | **A2A**: Key ID for RFC 8785 JCS canonicalization and JWS Agent Card signing. |
| `a2a.allowed_signing_algs` | None | `["ES256", "RS256", "EdDSA"]` | **A2A**: Permitted cryptographic algorithms for incoming Agent Cards. |
| `a2a.trusted_card_jwks` | None | `{}` | **A2A**: Map of upstream agent slug to JWKS URL for signature verification. |
| `files.storage_dir` | `GOROUTE_FILES_DIR` | `os.TempDir()/goroute-files` | **Files**: Local disk directory for stateful tenant assets and batch files (`0700` dir, `0600` file mode). |
| `providers.custom_openai_compatible[].id` | None | Required | **Custom Provider**: Unique string identifier (e.g. `"inhouse-vllm"`). |
| `providers.custom_openai_compatible[].display_name` | None | Required | **Custom Provider**: Human-readable label for logs and UI. |
| `providers.custom_openai_compatible[].base_url` | None | Required | **Custom Provider**: Target endpoint base URL (e.g. `http://vllm-cluster:8000/v1`). |
| `providers.custom_openai_compatible[].supports_streaming` | None | `"true"` | **Custom Provider**: Streaming capability override (`"true"`, `"false"`, `"unknown"`). |
| `providers.custom_openai_compatible[].supports_tools` | None | `"true"` | **Custom Provider**: Function/tool calling capability override. |
| `providers.custom_openai_compatible[].supports_structured_outputs` | None | `"true"` | **Custom Provider**: Structured outputs (`response_format: json_schema`) support. |
| `providers.custom_openai_compatible[].supports_list_models` | None | `true` | **Custom Provider**: Whether upstream endpoint implements `GET /models`. |
| `providers.custom_openai_compatible[].default_headers` | None | `{}` | **Custom Provider**: HTTP headers automatically attached to outbound calls. |
---
## 3. Model List & Root-Level Fallback Nesting
The `model_list` array defines the virtual routing topology presented to clients.
### Root-Level `fallbacks: [...]` Nesting Architecture
Each entry in `model_list` defines:
1. `model_name`: The virtual alias or model identifier requested by the client (e.g., `"fast"`, `"reasoning"`, `"gpt-4o"`).
2. `routing_strategy`: The balancing algorithm across sibling deployments (`"weighted-shuffle"`, `"least-busy"`, `"latency-based"`, `"usage-based-headroom"`, `"lowest-cost"`, `"round-robin"`).
3. `fallbacks`: **Explicit root-level array** of fallback model names triggered when this model group experiences an outage, exhausted retries, or rate limit rejections.
4. `model_params`: Upstream target configuration (`model`, `api_key`, `weight`, `rpm`, `tpm`, `azure_deployment_name`, etc.).
```mermaid
flowchart TD
Client["Client Request: model: 'gpt-4o'"] --> Primary["Primary Group: gpt-4o
(openai/gpt-4o)"]
Primary -->|"HTTP 429 / 5xx & Retries Exhausted"| FB1["Fallback 1: claude-3-5-sonnet
(anthropic/claude-3-5-sonnet)"]
FB1 -->|"HTTP 429 / 5xx & Retries Exhausted"| FB2["Fallback 2: gemini-2.5-pro
(gemini/gemini-2.5-pro)"]
FB2 -->|Success| Response["200 OK (X-Gateway-Provider: gemini)"]
```
### Declarative Example: Multi-Tier Failover
```yaml
model_list:
# Primary Tier: GPT-4o with cross-cloud fallbacks
- model_name: "gpt-4o"
routing_strategy: "weighted-shuffle"
fallbacks:
- "claude-3-5-sonnet"
- "gemini-2.5-pro"
model_params:
model: "openai/gpt-4o"
api_key: "os.environ/OPENAI_API_KEY"
weight: 100
rpm: 1000
tpm: 80000
# Fallback Tier 1: Claude 3.5 Sonnet
- model_name: "claude-3-5-sonnet"
routing_strategy: "latency-based"
fallbacks:
- "gemini-2.5-pro"
model_params:
model: "anthropic/claude-3-5-sonnet-20241022"
api_key: "os.environ/ANTHROPIC_API_KEY"
weight: 100
# Fallback Tier 2: Gemini 2.5 Pro
- model_name: "gemini-2.5-pro"
routing_strategy: "weighted-shuffle"
model_params:
model: "gemini/gemini-2.5-pro"
api_key: "os.environ/GEMINI_API_KEY"
weight: 100
# High-Throughput "fast" Group: Multi-Provider Load Balancing
- model_name: "fast"
routing_strategy: "least-busy"
fallbacks:
- "fast-backup"
model_params:
model: "gemini/gemini-2.5-flash"
api_key: "os.environ/GEMINI_API_KEY"
weight: 70
- model_name: "fast"
routing_strategy: "least-busy"
model_params:
model: "openai/gpt-4o-mini"
api_key: "os.environ/OPENAI_API_KEY"
weight: 30
# Emergency Local Backup
- model_name: "fast-backup"
routing_strategy: "weighted-shuffle"
model_params:
model: "ollama/llama3.3"
api_base: "http://localhost:11434"
weight: 100
```
> [!IMPORTANT]
> The `fallbacks: [...]` slice is declared at the root level of each `model_list` entry alongside `model_name` and `model_params`. It must never be nested inside the `model_params` object. The router traverses fallbacks strictly in listed order, respecting circuit breaker states for each candidate before dispatching.
---
## Deployment and Storage Architecture
URL: https://goroute-ai.github.io/docs/deploy/deployment-and-storage
Markdown: https://goroute-ai.github.io/docs/deploy/deployment-and-storage.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Deployment and Storage Architecture
GoRoute AI Gateway supports two operational storage modes:
1. **In-Memory Mode (`storage.mode: memory`)**: Zero external dependencies. Ideal for local unit testing, CI pipelines, and ephemeral container runs. Secrets are kept in-process in an AES-256-GCM vault, but state does not survive process restarts.
2. **Clustered PostgreSQL + Redis (`storage.mode: postgres`)**: The enterprise production topology. PostgreSQL provides ACID durability for multi-tenant organizations, model catalogs, encrypted credentials, virtual keys, audit trails, stateful files, and asynchronous batch jobs. Redis (`go-redis` `UniversalClient`) supports standalone, Sentinel (`storage.redis_master_name`), and Cluster (`storage.redis_addrs`) modes for distributed L2 key caching, atomic sliding-window rate limiting, response caching, circuit breaker state, and real-time Pub/Sub streams.
For production Kubernetes deployments, choose between the [Helm Chart Deployment Guide](https://goroute-ai.github.io/docs/deploy/helm-chart) (`deployments/helm/goroute/`) and the [Kubernetes Kustomize Deployment Guide](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize) (`deployments/k8s/`).
```mermaid
flowchart TD
subgraph Clients["Inference & Management Clients"]
HTTPClients["OpenAI / Anthropic SDKs"]
OperatorUI["Operator Web UI (/ui/)"]
AgentClients["MCP / A2A Clients"]
end
subgraph Cluster["Gateway Cluster Pods"]
Pod1["GoRoute Pod 1"]
Pod2["GoRoute Pod 2"]
end
subgraph Storage["Clustered Persistence Layer"]
PG[("PostgreSQL 16+
(16 Persistent Tables & Migrations)")]
Redis[("Redis 7+
(L2 Caching, Rate Limiting, Pub/Sub)")]
end
Clients --> Pod1
Clients --> Pod2
Pod1 --> PG
Pod2 --> PG
Pod1 <--> Redis
Pod2 <--> Redis
```
---
## 1. PostgreSQL Schema Migrations (0001 to 0015)
The gateway automatically manages its database schema at startup using embedded SQL migration scripts located in `migrations/`. Migrations are applied sequentially inside an atomic transaction (note that `0009` does not exist; numbering advances from `0008` directly to `0010`).
| Migration File | Description | Core Operations |
| :--- | :--- | :--- |
| **`0001_init_schema.up.sql`** | Core Schema Initialization | Enables `pgcrypto`. Creates base multi-tenant tables (`global_models`, `organizations`, `org_provider_subscriptions`, `org_model_policies`, `teams`, `users`, `virtual_keys`, `provider_credentials`, `model_groups`, `model_deployments`, `spend_logs`, `audit_logs`). |
| **`0002_add_platform_modules.up.sql`** | Platform Governance & Guardrails | Introduces `model_pricing_history` (immutable audit log of token pricing changes) and `guardrail_policies` (PII masking and prompt injection rules). |
| **`0003_add_mcp_and_a2a.up.sql`** | MCP Hub & A2A Network Tables | Creates MCP entity tables (`mcp_servers`, `mcp_tools`, `mcp_toolsets`, `mcp_user_credentials`) and A2A tables (`a2a_agents`, `a2a_tasks`, `a2a_task_events`, `a2a_stream_publications`, `a2a_outbox_events`, `a2a_push_notification_configs`). |
| **`0004_add_pricing_status.up.sql`** | Pricing Provenance & Azure Fields | Adds `pricing_status`, `pricing_provenance_source`, and `pricing_effective_date` to `global_models` and `spend_logs`. Adds `azure_deployment_name` and `api_version` to `provider_credentials`. Backfills audited seed models. |
| **`0005_add_model_capabilities.up.sql`** | Persistent Capabilities & Provenance | Adds `capabilities` JSONB column to `global_models` for tri-state capabilities tracking. Adds `pricing_status` and `pricing_provenance_source` to `model_pricing_history`. |
| **`0006_add_supports_inference.up.sql`** | Non-Inference Product Segregation | Adds `supports_inference` boolean flag to `global_models` to distinguish callable LLM deployments from ancillary catalog products. |
| **`0007_add_modality_pricing_and_spend.up.sql`** | Multi-Modal Spend & Batch Assets | Adds modality pricing columns (`rerank_cost_micros_per_search_unit`, `image_cost_micros_base`, `audio_stt_cost_micros_per_second`, etc.) to `global_models`, adds `modality` and `modality_units` JSONB to `spend_logs`, and creates `files` (`file_uploads`) and `batches` tables. |
| **`0008_add_spend_event_outbox.up.sql`** | Durable Spend Event Outbox | Creates `spend_event_outbox` (`pending`, `processing`, `delivered`, `dead_letter`) with retry/lock indexes for durable, idempotent Redis/PostgreSQL spend reconciliation. |
| **`0010_guardrail_mode.up.sql`** | `logging_only` Guardrails & All-Role Scanning | Adds `mode` (`enforce` / `logging_only`) and `scan_all_roles` columns to `guardrail_policies`. *(Note: `0009` does not exist.)* |
| **`0011_budget_reset_outbox.up.sql`** | Budget Reset Outbox | Creates `budget_reset_outbox` (`org`, `team`, `key` scopes) so expired budget period rollovers in PostgreSQL reliably clear Redis spend counters. |
| **`0012_add_cache_write_and_operations.up.sql`** | Cache-Write Pricing & Operations | Adds `cache_write_cost_micros_per_1m` and `supported_operations` JSONB to `global_models` for prompt cache-write FinOps and `SupportedOperations` routing enforcement. *(v1.6.0 also removes invalid `::uuid` casts on `VARCHAR(64)` `org_id`/`team_id`/`user_id` columns in `file_uploads` queries.)* |
| **`0013_user_limits_and_spend_indexes.up.sql`** | Per-User Limits & Spend Indexes | Adds `rpm_limit`, `tpm_limit`, `max_concurrent_requests`, `max_budget_micros`, `spent_micros`, `budget_reset_period`, and `budget_reset_at` to `users`, plus `(org_id, team_id)` and `(org_id, user_id)` indexes on `spend_logs`. |
| **`0014_budget_reset_outbox_user_scope.up.sql`** | User Budget Reset Scope | Updates `budget_reset_outbox_scope_check` to allow `'user'` alongside `'org'`, `'team'`, and `'key'`. |
| **`0015_team_org_concurrency.up.sql`** | Org & Team Concurrency Caps | Adds `max_concurrent_requests INTEGER NOT NULL DEFAULT 0` to `organizations` and `teams`. |
---
## 2. The 16 Persistent Table Schemas
### 1. `global_models`
Stores the unified catalog of all discovered and manually registered LLMs across providers.
```sql
CREATE TABLE global_models (
id VARCHAR(128) PRIMARY KEY, -- e.g. "openai/gpt-4o", "gemini/gemini-2.5-flash"
provider VARCHAR(32) NOT NULL,
model_name VARCHAR(128) NOT NULL,
display_name VARCHAR(255) NOT NULL,
max_input_tokens INTEGER NOT NULL DEFAULT 128000,
max_output_tokens INTEGER NOT NULL DEFAULT 16384,
supports_streaming BOOLEAN NOT NULL DEFAULT TRUE,
supports_tools BOOLEAN NOT NULL DEFAULT TRUE,
supports_vision BOOLEAN NOT NULL DEFAULT FALSE,
supports_reasoning BOOLEAN NOT NULL DEFAULT FALSE,
supports_embeddings BOOLEAN NOT NULL DEFAULT FALSE,
supports_inference BOOLEAN NOT NULL DEFAULT TRUE,
capabilities JSONB NOT NULL DEFAULT '{}'::jsonb,
input_cost_micros_per_1m BIGINT NOT NULL DEFAULT 0,
output_cost_micros_per_1m BIGINT NOT NULL DEFAULT 0,
cached_input_cost_micros_per_1m BIGINT NOT NULL DEFAULT 0,
reasoning_cost_micros_per_1m BIGINT NOT NULL DEFAULT 0,
rerank_cost_micros_per_search_unit BIGINT DEFAULT 0,
image_cost_micros_base BIGINT DEFAULT 0,
image_cost_micros_hd BIGINT DEFAULT 0,
audio_stt_cost_micros_per_second BIGINT DEFAULT 0,
audio_tts_cost_micros_per_1k_chars BIGINT DEFAULT 0,
modality_pricing_json JSONB DEFAULT '{}'::jsonb,
pricing_status VARCHAR(32) NOT NULL DEFAULT 'unknown',
pricing_provenance_source VARCHAR(255),
pricing_effective_date VARCHAR(64),
lifecycle_status VARCHAR(32) NOT NULL DEFAULT 'active',
deprecation_date TIMESTAMPTZ,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
discovered_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (provider, model_name)
);
```
### 2. `organizations`
Multi-tenant root container managing corporate spend caps, RPM/TPM ceilings, and billing cycles.
```sql
CREATE TABLE organizations (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
name VARCHAR(255) NOT NULL,
slug VARCHAR(128) NOT NULL UNIQUE,
max_budget_micros BIGINT NOT NULL DEFAULT 0, -- 0 = unlimited
spent_micros BIGINT NOT NULL DEFAULT 0,
budget_reset_period VARCHAR(32) NOT NULL DEFAULT 'monthly', -- 'daily', 'monthly', 'never'
budget_reset_at TIMESTAMPTZ,
rpm_limit INTEGER NOT NULL DEFAULT 0,
tpm_limit INTEGER NOT NULL DEFAULT 0,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 3. `org_provider_subscriptions`
Layer 2 governance controlling which upstream cloud providers an organization is permitted to call.
```sql
CREATE TABLE org_provider_subscriptions (
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
provider VARCHAR(32) NOT NULL,
is_enabled BOOLEAN NOT NULL DEFAULT TRUE,
auto_enable_new BOOLEAN NOT NULL DEFAULT FALSE,
subscribed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (org_id, provider)
);
```
### 4. `org_model_policies`
Layer 3 governance allowing or forbidding specific catalog models within an organization.
```sql
CREATE TABLE org_model_policies (
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
global_model_id VARCHAR(128) NOT NULL REFERENCES global_models(id) ON DELETE CASCADE,
is_enabled BOOLEAN NOT NULL DEFAULT TRUE,
approved_by VARCHAR(255) NOT NULL DEFAULT 'system',
approved_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (org_id, global_model_id)
);
```
### 5. `teams`
Sub-organizational entities representing engineering squads, divisions, or departments.
```sql
CREATE TABLE teams (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
name VARCHAR(255) NOT NULL,
max_budget_micros BIGINT NOT NULL DEFAULT 0,
spent_micros BIGINT NOT NULL DEFAULT 0,
budget_reset_period VARCHAR(32) NOT NULL DEFAULT 'monthly',
budget_reset_at TIMESTAMPTZ,
rpm_limit INTEGER NOT NULL DEFAULT 0,
tpm_limit INTEGER NOT NULL DEFAULT 0,
allowed_models TEXT[] NOT NULL DEFAULT '{}',
is_active BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (org_id, name)
);
```
### 6. `users`
Individual human operators or service accounts bound to organizations and teams.
```sql
CREATE TABLE users (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
team_id VARCHAR(64) REFERENCES teams(id) ON DELETE SET NULL,
email VARCHAR(255) NOT NULL,
role VARCHAR(32) NOT NULL DEFAULT 'member', -- 'admin', 'member', 'read_only'
is_active BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (org_id, email)
);
```
### 7. `virtual_keys`
Client authentication bearer tokens with rate limits, budgets, and model allowlists.
```sql
CREATE TABLE virtual_keys (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
team_id VARCHAR(64) REFERENCES teams(id) ON DELETE SET NULL,
user_id VARCHAR(64) REFERENCES users(id) ON DELETE SET NULL,
name VARCHAR(255) NOT NULL,
key_hash VARCHAR(64) NOT NULL UNIQUE, -- SHA-256 hash of secret part
key_prefix VARCHAR(16) NOT NULL, -- Display prefix e.g. "sk-gw-a1b2"
allowed_models TEXT[] NOT NULL DEFAULT '{"*"}',
allowed_endpoints TEXT[] NOT NULL DEFAULT '{"*"}',
max_budget_micros BIGINT NOT NULL DEFAULT 0,
spent_micros BIGINT NOT NULL DEFAULT 0,
rpm_limit INTEGER NOT NULL DEFAULT 0,
tpm_limit INTEGER NOT NULL DEFAULT 0,
expires_at TIMESTAMPTZ,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 8. `provider_credentials`
AES-256-GCM encrypted storage for upstream provider API keys and cloud service account credentials.
```sql
CREATE TABLE provider_credentials (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
provider VARCHAR(32) NOT NULL,
name VARCHAR(255) NOT NULL,
encrypted_api_key TEXT NOT NULL,
key_nonce VARCHAR(64) NOT NULL,
key_hash VARCHAR(64) NOT NULL,
aws_access_key_id VARCHAR(128),
encrypted_aws_secret TEXT,
aws_region VARCHAR(32),
encrypted_gcp_sa_json TEXT,
gcp_project_id VARCHAR(128),
gcp_region VARCHAR(32),
azure_deployment_name VARCHAR(255),
api_version VARCHAR(64),
custom_endpoint_url TEXT,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (org_id, provider, name)
);
```
### 9. `model_groups`
Logical routing targets aggregating sibling deployments under a shared virtual alias (`"fast"`, `"reasoning"`).
```sql
CREATE TABLE model_groups (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
name VARCHAR(128) NOT NULL,
routing_strategy VARCHAR(32) NOT NULL DEFAULT 'weighted_shuffle',
fallback_group_id VARCHAR(64) REFERENCES model_groups(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (org_id, name)
);
```
### 10. `model_deployments`
Individual physical provider routing targets bound to credentials with health and circuit breaker tracking.
```sql
CREATE TABLE model_deployments (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
model_group_id VARCHAR(64) NOT NULL REFERENCES model_groups(id) ON DELETE CASCADE,
global_model_id VARCHAR(128) NOT NULL REFERENCES global_models(id) ON DELETE CASCADE,
credential_id VARCHAR(64) NOT NULL REFERENCES provider_credentials(id) ON DELETE RESTRICT,
deployment_name VARCHAR(255) NOT NULL,
weight INTEGER NOT NULL DEFAULT 100,
priority INTEGER NOT NULL DEFAULT 0,
rpm_limit INTEGER NOT NULL DEFAULT 0,
tpm_limit INTEGER NOT NULL DEFAULT 0,
circuit_breaker_mode VARCHAR(32) NOT NULL DEFAULT 'standard',
status VARCHAR(32) NOT NULL DEFAULT 'active', -- 'active', 'degraded', 'cooldown', 'disabled'
consecutive_failures INTEGER NOT NULL DEFAULT 0,
cooldown_until TIMESTAMPTZ,
last_failure_at TIMESTAMPTZ,
last_success_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 11. `spend_logs`
High-throughput FinOps ledger recording every billable inference operation in integer micro-dollars.
```sql
CREATE TABLE spend_logs (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
team_id VARCHAR(64) REFERENCES teams(id) ON DELETE SET NULL,
user_id VARCHAR(64) REFERENCES users(id) ON DELETE SET NULL,
virtual_key_id VARCHAR(64) REFERENCES virtual_keys(id) ON DELETE SET NULL,
deployment_id VARCHAR(64) REFERENCES model_deployments(id) ON DELETE SET NULL,
global_model_id VARCHAR(128) NOT NULL,
provider VARCHAR(32) NOT NULL,
modality VARCHAR(32) NOT NULL DEFAULT 'chat',
prompt_tokens INTEGER NOT NULL DEFAULT 0,
completion_tokens INTEGER NOT NULL DEFAULT 0,
cached_prompt_tokens INTEGER NOT NULL DEFAULT 0,
reasoning_tokens INTEGER NOT NULL DEFAULT 0,
total_tokens INTEGER NOT NULL DEFAULT 0,
modality_units JSONB DEFAULT '{}'::jsonb,
cost_micros BIGINT NOT NULL DEFAULT 0,
request_duration_ms INTEGER NOT NULL DEFAULT 0,
status_code INTEGER NOT NULL DEFAULT 200,
error_code VARCHAR(64),
pricing_status VARCHAR(32) NOT NULL DEFAULT 'unrecorded',
pricing_provenance_source VARCHAR(255),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 12. `audit_logs`
Tamper-resistant audit log capturing administrative actions, policy mutations, and credential access.
```sql
CREATE TABLE audit_logs (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
user_id VARCHAR(64),
actor VARCHAR(255) NOT NULL,
action VARCHAR(128) NOT NULL,
resource VARCHAR(128) NOT NULL,
resource_id VARCHAR(128) NOT NULL,
changes JSONB NOT NULL DEFAULT '{}'::jsonb,
ip_address VARCHAR(64),
user_agent TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 13. `model_pricing_history`
Immutable historical log of every catalog token and modality pricing update.
```sql
CREATE TABLE model_pricing_history (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
global_model_id VARCHAR(128) NOT NULL REFERENCES global_models(id) ON DELETE CASCADE,
input_cost_micros_per_1m BIGINT NOT NULL,
output_cost_micros_per_1m BIGINT NOT NULL,
cached_input_cost_micros_per_1m BIGINT NOT NULL DEFAULT 0,
reasoning_cost_micros_per_1m BIGINT NOT NULL DEFAULT 0,
pricing_status VARCHAR(32) NOT NULL DEFAULT 'unknown',
pricing_provenance_source VARCHAR(255),
effective_from TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 14. `guardrail_policies`
Content safety rules, PII detectors, and prompt injection filters attached to tenants.
```sql
CREATE TABLE guardrail_policies (
id VARCHAR(64) PRIMARY KEY DEFAULT gen_random_uuid()::text,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
name VARCHAR(255) NOT NULL,
is_active BOOLEAN NOT NULL DEFAULT TRUE,
rules JSONB NOT NULL DEFAULT '[]'::jsonb,
action VARCHAR(32) NOT NULL DEFAULT 'block', -- 'block', 'mask', 'warn'
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
```
### 15. `files`
Metadata and disk pointers for stateful tenant assets (fine-tuning datasets, Whisper audio, batch inputs).
```sql
CREATE TABLE files (
id VARCHAR(64) PRIMARY KEY,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
team_id VARCHAR(64) REFERENCES teams(id) ON DELETE SET NULL,
user_id VARCHAR(64) REFERENCES users(id) ON DELETE SET NULL,
bytes BIGINT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
filename VARCHAR(255) NOT NULL,
purpose VARCHAR(64) NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'uploaded',
content_sha256 VARCHAR(64) NOT NULL,
storage_path TEXT NOT NULL,
upstream_provider VARCHAR(64),
upstream_id VARCHAR(128),
expires_at TIMESTAMPTZ
);
```
### 16. `batches`
Asynchronous inference batch requests evaluated within 24-hour turnaround windows for 50% discount.
```sql
CREATE TABLE batches (
id VARCHAR(64) PRIMARY KEY,
org_id VARCHAR(64) NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
team_id VARCHAR(64) REFERENCES teams(id) ON DELETE SET NULL,
input_file_id VARCHAR(64) NOT NULL REFERENCES files(id),
output_file_id VARCHAR(64) REFERENCES files(id),
error_file_id VARCHAR(64) REFERENCES files(id),
endpoint VARCHAR(128) NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'validating',
completion_window VARCHAR(32) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ,
metadata JSONB DEFAULT '{}'::jsonb,
request_counts JSONB DEFAULT '{"total":0,"completed":0,"failed":0}'::jsonb,
upstream_provider VARCHAR(64),
upstream_id VARCHAR(128)
);
```
---
## 3. Redis Cluster, Sentinel & Keyspace Matrix
`RedisFabric` (`internal/storage/redis.go`) wraps a `go-redis` `UniversalClient` protected by a circuit breaker (with `redis.Nil` cache misses treated as non-failures). It supports three deployment topologies:
- **Standalone**: `storage.redis_addr` (`GATEWAY_REDIS_ADDR`)
- **Redis Cluster**: `storage.redis_addrs` (`GATEWAY_REDIS_ADDRS`, comma-separated seed nodes)
- **Redis Sentinel**: `storage.redis_addrs` combined with `storage.redis_master_name` (`GATEWAY_REDIS_MASTER_NAME`)
All multi-key Lua scripts for rate limiting, budget reservations, and spend reconciliation use `{org:}` hash tags (e.g., `gw:spend:{org:}:key:{keyID}`, `gw:budget:{org:}:reservations:key:{keyID}`) so every key belonging to an organization maps to the same Redis Cluster hash slot.
All keys are namespaced with the `gw:` prefix.
| Redis Key Pattern | Type | Configured TTL | Subsystem & Purpose |
| :--- | :--- | :--- | :--- |
| `gw:auth:key:{keyHash}` | String | `auth.l2_cache_ttl` *(default `5m`)* | **Auth L2 Cache**: Serialized `IdentityContext` preventing database lookups on every request. |
| `gw:rl:rpm:key:{keyID}:{minute}` | String (Int) | `window_seconds * 2` *(120s)* | **Rate Limiter**: Sliding-minute requests counter for virtual key. |
| `gw:rl:tpm:key:{keyID}:{minute}` | String (Int) | `window_seconds * 2` *(120s)* | **Rate Limiter**: Sliding-minute tokens counter for virtual key. |
| `gw:rl:conc:{keyID}` | String (Int) | `rate_limit.concurrency_ttl_sec` *(300s)* | **Rate Limiter**: Atomic lease counter tracking concurrent in-flight requests. |
| `gw:spend:key:{keyID}` | String (Int) | None *(Persistent Counter)* | **FinOps**: Fast in-memory spend accumulator for virtual key. |
| `gw:rl:rpm:team:{teamID}:{minute}` | String (Int) | `window_seconds * 2` *(120s)* | **Rate Limiter**: Team-level sliding-minute requests counter. |
| `gw:rl:tpm:team:{teamID}:{minute}` | String (Int) | `window_seconds * 2` *(120s)* | **Rate Limiter**: Team-level sliding-minute tokens counter. |
| `gw:spend:team:{teamID}` | String (Int) | None *(Persistent Counter)* | **FinOps**: Team-level spend accumulator. |
| `gw:rl:rpm:org:{orgID}:{minute}` | String (Int) | `window_seconds * 2` *(120s)* | **Rate Limiter**: Organization-level sliding-minute requests counter. |
| `gw:rl:tpm:org:{orgID}:{minute}` | String (Int) | `window_seconds * 2` *(120s)* | **Rate Limiter**: Organization-level sliding-minute tokens counter. |
| `gw:spend:org:{orgID}` | String (Int) | None *(Persistent Counter)* | **FinOps**: Organization-level spend accumulator. |
| `gw:cache:resp:{sha256(req)}` | String (JSON) | Configured Cache TTL *(default `1h`)* | **Response Cache**: Exact-match cached `CanonicalChatResponse`. |
| `gw:router:cooldown:{depID}` | String | `router.default_cooldown` *(default `60s`)* | **Circuit Breaker**: Cooldown lock asserting deployment is in `OPEN` state. |
| `gw:lock:discovery` | String | `60s` | **Discovery Worker**: Distributed leader election lock preventing duplicate sync sweeps. |
| `gw:mcp:hdl:{handle}` | String (JSON) | `mcp.handle_ttl` *(default `1h`)* | **MCP Discovery**: 256-bit opaque tool handle record binding. |
| `gw:events:invalidation` | Pub/Sub Channel | N/A | **Cluster Invalidation**: Broadcasts L1 cache eviction events when keys or policies mutate. |
| `gw:a2a:events:{taskID}` | Pub/Sub Channel | N/A | **A2A Streaming**: Real-time event bus broadcasting SSE updates to connected subscribers. |
> [!TIP]
> In production environments with multiple gateway pods, always configure Redis so that virtual key revocations and policy modifications broadcast via `gw:events:invalidation` take effect across all pods within milliseconds.
---
## Kubernetes & Kustomize Deployment
URL: https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize
Markdown: https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Kubernetes & Kustomize Deployment
GoRoute AI Gateway ships production-hardened Kubernetes manifests under [`deployments/k8s/`](https://github.com/goroute-ai/ai-gateway-go/tree/main/deployments/k8s) managed via Kustomize. The base overlay deploys a 2-replica stateless gateway tier backed by PostgreSQL 16 and Redis 7, with non-root container hardening, read-only root filesystems, and `/readyz` + `/healthz` probes. For parameterized production releases with HPA, PDB, NetworkPolicy, Ingress, and pre-upgrade migration hooks, see [Helm Chart Deployment](https://goroute-ai.github.io/docs/deploy/helm-chart).
```mermaid
flowchart TD
Ingress["Ingress / Load Balancer"] --> Svc["Service: gateway (ClusterIP :8080)"]
Svc --> Pod1["Pod: gateway-1
non-root UID 65532, readOnlyRootFS"]
Svc --> Pod2["Pod: gateway-2
non-root UID 65532, readOnlyRootFS"]
Pod1 --> PGSvc["Service: postgres (:5432)
PostgreSQL 16"]
Pod2 --> PGSvc
Pod1 <--> RedisSvc["Service: redis (:6379)
Redis 7 / Cluster / Sentinel"]
Pod2 <--> RedisSvc
Prom["Prometheus Scraper"] -->|Bearer Admin Key
GET /metrics| Svc
```
---
## 1. Kustomize Manifest Inventory
The `deployments/k8s/kustomization.yaml` file composes six resource manifests into a single deployable unit:
| File | Kind | Name | Purpose |
| :--- | :--- | :--- | :--- |
| `namespace.yaml` | `Namespace` | `goroute-test` | Isolates gateway, database, and cache resources (override in production overlays). |
| `postgres.yaml` | `Deployment`, `Service` | `postgres` | PostgreSQL 16 (`postgres:16-alpine`) on port `5432` with `pg_isready` readiness probe. |
| `redis.yaml` | `Deployment`, `Service` | `redis` | Redis 7 (`redis:7-alpine`) on port `6379` with `redis-cli ping` readiness probe. |
| `secret.example.yaml` | `Secret` | `gateway-secrets` | Template secret injecting `GATEWAY_POSTGRES_DSN`, `GATEWAY_MASTER_ADMIN_KEY`, and `GATEWAY_MASTER_ENCRYPTION_KEY`. |
| `gateway-deployment.yaml` | `Deployment` | `gateway` | 2-replica GoRoute AI Gateway (`goroute/ai-gateway`) with hardened security context and probes. |
| `service.yaml` | `Service` | `gateway` | Exposes port `8080` (`http`) across the `app: gateway` replica set. |
---
## 2. Applying the Manifests
Preview the rendered manifests before applying:
```bash
kubectl kustomize deployments/k8s/
```
Deploy to your cluster:
```bash
kubectl apply -k deployments/k8s/
kubectl -n goroute-test rollout status deployment/gateway --timeout=120s
```
To verify the multi-replica Kubernetes smoke suite locally against a `kind` or `k3d` cluster, run:
```bash
make test-k8s-smoke
```
:::warning Replace Example Secrets Before Production
`deployments/k8s/secret.example.yaml` contains placeholder test credentials. Never apply example secrets to a production cluster. Use External Secrets Operator, Sealed Secrets, or a Kustomize `secretGenerator` overlay backed by your KMS/Vault.
:::
---
## 3. Container Security Context & Probes
`deployments/k8s/gateway-deployment.yaml` enforces strict pod and container hardening:
- **Pod `securityContext`**: `runAsNonRoot: true`, `runAsUser: 65532` (distroless non-root user).
- **Container `securityContext`**:
- `allowPrivilegeEscalation: false`
- `readOnlyRootFilesystem: true`
- `capabilities.drop: ["ALL"]`
- **Graceful Shutdown**: `terminationGracePeriodSeconds: 30` paired with `server.shutdown_timeout` (`15s` default) to drain active SSE streams.
- **Resource Requests & Limits**: Requests `100m` CPU / `128Mi` memory; limits `500m` CPU / `512Mi` memory (tune upward for high-concurrency streaming or large multipart file uploads).
- **Health Probes**:
- **Readiness (`GET /readyz` on `:8080`)**: `initialDelaySeconds: 3`, `periodSeconds: 5`. Verifies live PostgreSQL and Redis connectivity before routing traffic.
- **Liveness (`GET /healthz` on `:8080`)**: `initialDelaySeconds: 5`, `periodSeconds: 10`. Confirms process health without cascading restarts on transient dependency blips.
:::tip Writable Scratch Volume for Multipart File Uploads
If you enable `/v1/files` or audio multipart disk spooling (`POST /v1/audio/transcriptions`) with `readOnlyRootFilesystem: true`, mount an `emptyDir` volume at `/tmp` (or your configured temp/file storage path) so Go's multipart spooled files can write safely.
:::
---
## 4. Production Overlay: Namespace, Secrets & Image Tag
Create a production Kustomize overlay (e.g., `deployments/k8s/overlays/prod/kustomization.yaml`) rather than editing the base manifests directly:
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: goroute-prod
resources:
- ../../
images:
- name: goroute/ai-gateway
newName: ghcr.io/goroute-ai/ai-gateway-go
newTag: v1.8.0
patches:
- target:
kind: Namespace
name: goroute-test
patch: |-
- op: replace
path: /metadata/name
value: goroute-prod
- target:
kind: Deployment
name: gateway
patch: |-
- op: add
path: /spec/template/spec/containers/0/env/-
value:
name: GATEWAY_ENV
value: "production"
```
---
## 5. PostgreSQL 16 & Redis Cluster / Sentinel
### PostgreSQL 16
The base `postgres.yaml` uses `postgres:16-alpine` with a single replica for smoke testing. For production:
- Use a managed PostgreSQL 16+ service (RDS, Cloud SQL, Neon) or a HA operator (CloudNativePG) with TLS (`?sslmode=verify-full`) in `GATEWAY_POSTGRES_DSN`.
- Embedded SQL migrations (`0001` through `0015`) run automatically at startup inside atomic transactions.
### Redis Cluster & Sentinel
GoRoute's `RedisFabric` uses `go-redis` `UniversalClient` with `{org:}` hash-tagged keys so multi-key Lua scripts for rate limiting and response caching execute atomically on a single slot:
| Topology | Environment Variables |
| :--- | :--- |
| **Standalone** | `GATEWAY_REDIS_ADDR=redis:6379` |
| **Redis Cluster** | `GATEWAY_REDIS_ADDRS=redis-0:6379,redis-1:6379,redis-2:6379` |
| **Redis Sentinel** | `GATEWAY_REDIS_ADDRS=sentinel-0:26379,sentinel-1:26379,sentinel-2:26379` + `GATEWAY_REDIS_MASTER_NAME=mymaster` |
| **Authentication** | `GATEWAY_REDIS_PASS=` |
---
## 6. Prometheus Metrics Scraping
The `GET /metrics` endpoint exposes Prometheus text format `0.0.4` metrics using bounded-cardinality reservoir sampling (`1,024` samples) and requires `Authorization: Bearer `.
Configure a Prometheus `ServiceMonitor` or `ScrapeConfig` referencing the admin bearer token secret:
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: goroute-gateway
namespace: goroute-prod
spec:
selector:
matchLabels:
app: gateway
endpoints:
- port: http
path: /metrics
interval: 15s
bearerTokenSecret:
name: gateway-secrets
key: GATEWAY_MASTER_ADMIN_KEY
```
See [Observability, Metrics & Health Probes](https://goroute-ai.github.io/docs/deploy/observability) for the full exported metric catalog and OTLP tracing configuration.
---
## Helm Chart Deployment
URL: https://goroute-ai.github.io/docs/deploy/helm-chart
Markdown: https://goroute-ai.github.io/docs/deploy/helm-chart.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Helm Chart Deployment
GoRoute AI Gateway ships an official Kubernetes Helm chart (`goroute`, chart version `0.3.0`, app version `1.8.0`) under [`deployments/helm/goroute/`](https://github.com/goroute-ai/ai-gateway-go/tree/main/deployments/helm/goroute) and as a packaged `.tgz` archive on the GitHub Releases page.
For raw manifests without Helm, see [Kubernetes & Kustomize Deployment](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize).
---
## 1. Chart Overview & Optional Dependencies
`Chart.yaml` defines `apiVersion: v2` (`type: application`) with two optional Bitnami subchart dependencies:
| Dependency | Version Constraint | Condition Key | Default |
| :--- | :--- | :--- | :--- |
| `postgresql` | `16.x.x` (`https://charts.bitnami.com/bitnami`) | `postgresql.enabled` | `false` |
| `redis` | `20.x.x` (`https://charts.bitnami.com/bitnami`) | `redis.enabled` | `false` |
In production, keep `postgresql.enabled: false` and `redis.enabled: false` when pointing to managed PostgreSQL 16+ and Redis 7 / Cluster / Sentinel services. For single-cluster evaluations, enable both subcharts via `--set postgresql.enabled=true --set redis.enabled=true` after updating dependencies.
---
## 2. Quick Install & Upgrade
### Install from Local Chart or Release Archive
```bash
# Option A: Install directly from a release .tgz archive
helm upgrade --install goroute ./goroute-0.3.0.tgz \
--namespace goroute-prod --create-namespace \
--set secrets.existingSecret="goroute-prod-secrets" \
--set migration.enabled=true
# Option B: Install from the cloned repository
helm dependency update deployments/helm/goroute
helm upgrade --install goroute deployments/helm/goroute \
--namespace goroute-prod --create-namespace \
-f prod-values.yaml
```
> [!WARNING]
> Default values in `values.yaml` include placeholder secrets (`changeme-master-admin-key`, sample `postgresDSN`, and sample `masterEncryptionKey`). Always set `secrets.existingSecret` (recommended) or override every `secrets.*` value via an external secret manager before deploying to production.
---
## 3. Production Security, Scaling & Networking Templates
| Template | Kind / API | Default Behavior |
| :--- | :--- | :--- |
| `templates/deployment.yaml` | `apps/v1` `Deployment` | Non-root `podSecurityContext` (`runAsNonRoot: true`, UID/GID/fsGroup `65532`), `readOnlyRootFilesystem: true`, `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`, writable `/tmp` `emptyDir`, `/healthz` & `/readyz` probes, `terminationGracePeriodSeconds: 30`. |
| `templates/service.yaml` | `v1` `Service` | `ClusterIP` on port `8080` (`http`). Exposes `grpc` port on both `Service` and `Deployment` when `service.grpcPort > 0`. |
| `templates/hpa.yaml` | `autoscaling/v2` `HorizontalPodAutoscaler` | Enabled by default (`minReplicas: 2`, `maxReplicas: 10`, CPU `75%`, Memory `80%`). When enabled, static `replicaCount` is omitted from `Deployment`. |
| `templates/pdb.yaml` | `policy/v1` `PodDisruptionBudget` | Enabled by default (`minAvailable: 1`; also supports `maxUnavailable` when `minAvailable` is removed). |
| `templates/networkpolicy.yaml` | `networking.k8s.io/v1` `NetworkPolicy` | Enabled by default; restricts ingress to `service.port` (and `service.grpcPort` when `> 0`, optionally scoped via `networkPolicy.ingressFrom`) and egress to DNS (`53` UDP/TCP), PostgreSQL (`5432`), Redis (`6379`), and HTTPS (`443`). |
| `templates/ingress.yaml` | `networking.k8s.io/v1` `Ingress` | Disabled by default (`ingress.enabled: false`); supports `className`, `annotations`, `hosts`, and `tls`. |
| `templates/migration-job.yaml` | `batch/v1` `Job` | Disabled by default (`migration.enabled: false`). Runs as a `pre-install,pre-upgrade` Helm hook (`hook-weight: "-5"`) executing `/app/ai-gateway -config /app/configs/config.yaml -migrate-only`. |
| `templates/secret.yaml` | `v1` `Secret` | Created only when `secrets.create: true` and `secrets.existingSecret` is empty. Injects `GATEWAY_POSTGRES_DSN`, `GATEWAY_MASTER_ADMIN_KEY`, `GATEWAY_MASTER_ENCRYPTION_KEY`, and optional `GATEWAY_JWT_SECRET` / `GATEWAY_REDIS_PASSWORD`. |
---
## 4. `values.yaml` Reference
| Key | Default | Description |
| :--- | :--- | :--- |
| `replicaCount` | `2` | Static pod count (used only when `autoscaling.enabled: false`). |
| `image.repository` | `ghcr.io/goroute-ai/ai-gateway-go` | Container image repository. |
| `image.tag` | `""` | Overrides `Chart.AppVersion` (`1.8.0`) when non-empty. |
| `image.pullPolicy` | `IfNotPresent` | Kubernetes image pull policy. |
| `service.type` | `ClusterIP` | Kubernetes Service type. |
| `service.port` | `8080` | HTTP data plane, `/admin/v1`, `/ui/`, and `/metrics` port. |
| `service.grpcPort` | `0` | Optional A2A gRPC port (`0` disables gRPC Service/container port). |
| `ingress.enabled` | `false` | Enables the `Ingress` manifest. |
| `env` | `GATEWAY_PORT: "8080"`, `GATEWAY_ENV: "production"`, `GATEWAY_STORAGE_MODE: "postgres"`, `GATEWAY_REDIS_ADDR: "redis:6379"` | Non-secret environment variables injected into the gateway and migration job pods. |
| `secrets.create` | `true` | Creates the Kubernetes `Secret` when `secrets.existingSecret` is empty. |
| `secrets.existingSecret` | `""` | Name of a pre-existing Secret containing `GATEWAY_POSTGRES_DSN`, `GATEWAY_MASTER_ADMIN_KEY`, `GATEWAY_MASTER_ENCRYPTION_KEY`, etc. |
| `resources` | `requests: 100m / 128Mi`, `limits: 1000m / 512Mi` | Gateway pod CPU and memory requests/limits. |
| `livenessProbe` | `GET /healthz` (`:8080`, `initialDelaySeconds: 5`, `periodSeconds: 10`) | Container liveness probe configuration. |
| `readinessProbe` | `GET /readyz` (`:8080`, `initialDelaySeconds: 3`, `periodSeconds: 5`) | Container readiness probe checking PostgreSQL and Redis connectivity. |
| `autoscaling.enabled` | `true` | Enables HPA (`minReplicas: 2`, `maxReplicas: 10`, CPU `75%`, Mem `80%`). |
| `podDisruptionBudget.enabled` | `true` | Enables PDB (`minAvailable: 1`). |
| `networkPolicy.enabled` | `true` | Enables Pod `NetworkPolicy` with `egressPorts: {postgres: 5432, redis: 6379, https: 443}`. |
| `migration.enabled` | `false` | Enables the `pre-install,pre-upgrade` `-migrate-only` Job (`backoffLimit: 3`, `activeDeadlineSeconds: 300`). |
---
## 5. Related Guides
- [Kubernetes & Kustomize Deployment](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize)
- [Deployment and Storage Architecture](https://goroute-ai.github.io/docs/deploy/deployment-and-storage)
- [Production Readiness Checklist](https://goroute-ai.github.io/docs/deploy/production-checklist)
---
## Embedded Operator Web UI
URL: https://goroute-ai.github.io/docs/deploy/operator-ui
Markdown: https://goroute-ai.github.io/docs/deploy/operator-ui.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Embedded Operator Web UI
GoRoute AI Gateway embeds a high-performance **Operator Web UI** compiled with React 19, TypeScript, Vite, and Tailwind CSS. The entire console is bundled directly into the compiled Go binary using Go's `//go:embed` directive, eliminating external frontend hosting, CDN dependencies, or CORS setup.
The console is accessible by default at:
```text
http://localhost:8080/ui/
```
> [!NOTE]
> The gateway automatically redirects non-trailing slash requests (`/ui`) to `/ui/` (HTTP 301) to ensure relative static asset resolution functions properly across reverse proxies and ingress controllers.
---
## 1. Architecture: Single-Binary Go Embedding
The Operator Web UI resides in the repository under `web/`. When built for production, Vite compiles the React SPA into static HTML, JavaScript, and CSS bundles located in `web/dist/`, which are mirrored to `internal/ui/dist/`.
```mermaid
flowchart TD
subgraph Browser["Browser Client"]
SPA["React 19 SPA (Vite / Tailwind)"]
end
subgraph Binary["Go Single Binary (`cmd/gateway`)"]
ChiRouter["Chi Router (`/ui/*`)"]
EmbedFS["//go:embed all:dist
(internal/ui/ui.go)"]
HTML5Fallback["HTML5 History API Fallback
(index.html)"]
DataPlane["Data Plane (`/v1/*`)"]
ControlPlane["Control Plane (`/admin/v1/*`)"]
end
SPA -->|HTTP Requests| ChiRouter
ChiRouter -->|Hashed Asset Request| EmbedFS
ChiRouter -->|"Client Route: /ui/finops"| HTML5Fallback
SPA -->|Inference Stream| DataPlane
SPA -->|Admin Ops & Metrics| ControlPlane
```
### Go `embed.FS` Integration (`internal/ui/ui.go`)
The Go HTTP handler utilizes `embed.FS` and `http.FileServer` with tailored caching headers and HTML5 History API fallback routing:
```go
package ui
import (
"embed"
"io"
"io/fs"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
)
//go:embed all:dist
var embeddedDist embed.FS
// DistFS returns the sub-filesystem for the embedded dist directory.
func DistFS() (fs.FS, error) {
return fs.Sub(embeddedDist, "dist")
}
// RegisterRoutes mounts the embedded operator dashboard on the Chi router at /ui.
func RegisterRoutes(r chi.Router) {
distFS, err := fs.Sub(embeddedDist, "dist")
if err != nil {
return
}
fileServer := http.FileServer(http.FS(distFS))
// Redirect /ui to /ui/ with trailing slash for proper relative asset resolution
r.Get("/ui", func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/ui/", http.StatusMovedPermanently)
})
// Serve static assets or fallback to index.html for SPA client-side routing
r.Get("/ui/*", func(w http.ResponseWriter, r *http.Request) {
trimmedPath := strings.TrimPrefix(r.URL.Path, "/ui")
if trimmedPath == "" || trimmedPath == "/" {
trimmedPath = "/index.html"
}
cleanPath := strings.TrimPrefix(trimmedPath, "/")
// If the specific file exists and is not index.html, serve it
if cleanPath != "index.html" {
if f, err := distFS.Open(cleanPath); err == nil {
_ = f.Close()
if strings.HasPrefix(trimmedPath, "/assets/") {
// Immutable caching for hashed assets
w.Header().Set("Cache-Control", "public, max-age=31536000, immutable")
}
http.StripPrefix("/ui", fileServer).ServeHTTP(w, r)
return
}
}
// HTML5 History API fallback: serve index.html for client-side routing
indexFile, err := distFS.Open("index.html")
if err != nil {
http.NotFound(w, r)
return
}
defer indexFile.Close()
stat, err := indexFile.Stat()
if err != nil {
http.NotFound(w, r)
return
}
w.Header().Set("Content-Type", "text/html; charset=utf-8")
w.Header().Set("Cache-Control", "no-cache")
http.ServeContent(w, r, "index.html", stat.ModTime(), indexFile.(io.ReadSeeker))
})
}
```
### HTML5 History API Client-Side Routing
The web application uses client-side routing (`/ui/playground`, `/ui/models`, `/ui/finops`, etc.). When a user bookmarks, reloads, or directly navigates to a deep URL:
1. Chi receives the request at `/ui/{route}`.
2. The router checks if a physical file matches `{route}`. If not, it executes the fallback block.
3. The server responds with `index.html` (HTTP 200, `Cache-Control: no-cache`), allowing the React client router to parse `window.location.pathname` and render the matching view without 404 errors.
4. Assets under `/ui/assets/*` contain cryptographic content hashes generated by Vite (e.g. `index-B1x48z.js`) and are served with `Cache-Control: public, max-age=31536000, immutable`.
---
## 2. Production Bundle Budget Validation
To ensure the embedded gateway binary remains lightweight and does not bloat memory footprint or container images, the frontend build enforces a strict **bundle budget under 400 KB uncompressed** across all compiled files.
Budget enforcement is automated via `web/scripts/check-budget.mjs`:
| Artifact Category | Uncompressed Budget Threshold | Gzip Budget Threshold |
| :--- | :--- | :--- |
| **Max Single JavaScript Chunk** | 150 KB max | 45 KB max |
| **Max Single CSS File** | 35 KB max | 8 KB max |
| **Total Production Distribution** | 380 KB max *(budget ceiling 400 KB)* | 110 KB max |
### Verifying Bundle Budgets
Run the build script from the repository root:
```bash
make ui-build
```
Or execute directly inside `web/`:
```bash
cd web
npm run build
node scripts/check-budget.mjs
```
Sample output:
```text
📊 Checking Production Frontend Bundle Budgets:
────────────────────────────────────────────────────────
✅ PASS assets/index-D7x92A.js : 142.18 KB (gzip: 41.34 KB)
✅ PASS assets/index-C3y81B.css : 28.64 KB (gzip: 6.12 KB)
✅ PASS index.html : 0.73 KB (gzip: 0.41 KB)
────────────────────────────────────────────────────────
Total Bundle Size: 171.55 KB (gzipped: 47.87 KB)
✨ All frontend bundle budget constraints satisfied!
```
> [!CAUTION]
> Adding heavy frontend dependencies (such as large component suites, heavy charting libraries, or date manipulation packages) that cause uncompressed bundle size to exceed 380 KB will immediately fail the CI verification pipeline and block release builds.
---
## 3. Deep Dive: 8 Production Views
The Operator Web UI is organized into 8 modular views providing end-to-end administration across data plane inference, routing, multi-tenant governance, and agentic protocols.
```mermaid
graph LR
UI["Operator Web UI (/ui/)"]
UI --> V1["1. Overview"]
UI --> V2["2. Streaming Playground"]
UI --> V3["3. Model Catalogue"]
UI --> V4["4. Virtual API Keys"]
UI --> V5["5. FinOps Spend"]
UI --> V6["6. Guardrail Simulator"]
UI --> V7["7. Upstream Credentials"]
UI --> V8["8. Agentic Hub Explorer"]
```
### View 1: Overview Dashboard (`OverviewView.tsx`)
The landing view provides real-time system observability and operational telemetry:
- **System Health & Storage Indicators**: Displays active status for PostgreSQL connection pool, Redis cache fabric, and upstream provider availability.
- **Real-Time Inference KPI Counters**: Total inference requests, error rates, cache-hit percentage, and aggregated micro-dollar spend.
- **Deployment Status Grid**: Real-time status badges for all active model deployments (Healthy, Degraded, Tripped / In Cooldown).
- **Recent Audit Trail**: Live log feed of configuration mutations, key creations, and administrative actions with operator attribution.
### View 2: Streaming Playground (`PlaygroundView.tsx`)
An interactive developer testbench for validating models, prompt templates, and streaming reliability without writing code:
- **Interactive Chat Canvas**: Multi-turn conversational interface supporting `user`, `assistant`, and `system` message sequencing.
- **Model Switcher**: Direct selection of model groups (e.g., `"fast"`, `"reasoning"`, `"coding"`) or provider-qualified model IDs. Use the current model inventory for the signed-in key; examples elsewhere in this guide do not define the deployment's available models.
- **Hyperparameter Controls**: Sliders for temperature ([0.0, 2.0]), max completion tokens, top-p, frequency penalty, and presence penalty.
- **Real-Time Token-by-Token SSE Stream**: Consumes `text/event-stream` chunks via native Fetch API and Web Streams, rendering live markdown tokens.
- **Inference Diagnostics**: Real-time display of Time-To-First-Token (TTFT in milliseconds), total generation time, prompt/completion token counts, and micro-dollar cost.
### View 3: Model Catalogue Browser (`ModelsView.tsx`)
A searchable directory of model records registered or discovered in this gateway deployment. It is not a universal or necessarily exhaustive list of provider models; availability to callers is further filtered by organization policies and virtual-key permissions.
- **4-Layer Governance Visibility**: Inspects the hierarchical model availability status (Global Discovery -> Org Subscription -> Org Policy Override -> Virtual Key allowlist).
- **Capability Matrix**: Filter models by supported modalities:
- Streaming, Tool Calling, Vision, Reasoning / Thinking, Structured Outputs (`json_schema`), Dense Embeddings, Reranking, Audio STT / TTS, Stateful Batches, and Realtime WebSockets.
- **Pricing Transparency**: Real-time display of token pricing:
- Input tokens per 1M (USD per 1M), Output tokens per 1M (USD per 1M), Cached prompt tokens per 1M (USD per 1M), and Reasoning tokens per 1M (USD per 1M).
- **Model Provenance Badges**: Verified pricing provenance indicators (`verified`, `catalog-default`, or `custom-override`).
### View 4: Virtual API Keys (`KeysView.tsx`)
Multi-tenant credential and access governance for engineering teams and client applications:
- **Key Generation & Secret Hashing**: Issues high-entropy client keys (`sk-gw--`). The gateway stores only the cryptographic SHA-256 hash; the plaintext secret is revealed only once upon creation.
- **Tenant Scoping**: Assign keys to Organizations, Teams, and Users with metadata tags.
- **Rate & Budget Limits**: Configures Requests Per Minute (RPM), Tokens Per Minute (TPM), and hard micro-dollar spending budgets per key.
- **Model Whitelists & Wildcards**: Restricts keys to explicit model aliases (e.g. `["fast", "coding"]`) or wildcards (`["*"]`).
- **Instant Revocation & Rotation**: One-click key disablement or deletion with automated Redis Pub/Sub L1/L2 cache invalidation broadcast across all gateway pods.
### View 5: FinOps Spend Analytics (`FinOpsView.tsx`)
Enterprise cost accounting and budget management powered by the micro-dollar ledger:
- **Organization & Team Spend Aggregation**: Visual breakdown of total incurred spend against defined monthly/daily budgets.
- **Modality Cost Breakdown**: Detailed spend categorization across:
- `chat` (standard completions & thinking models)
- `rerank` (search units via Cohere / Jina)
- `audio` (Whisper STT duration seconds & TTS characters)
- `image` (DALL-E & Imagen generation units)
- `batches` (50% discounted batch jobs)
- **Spend Velocity & Projections**: Real-time spend rate monitoring to forecast end-of-month budget utilization.
- **Budget Alerts**: Visual indicators when tenants cross 80% soft warning or 100% hard blocking thresholds.
### View 6: Guardrail Simulator (`GuardrailsView.tsx`)
A staging simulator for testing security policies and content filters before applying them to production traffic:
- **Simulation Canvas**: Enter arbitrary test prompts to evaluate against configured guardrail policies in dry-run mode.
- **Detection Evaluators**:
- **Prompt Injection & Jailbreak Detection**: Evaluates heuristic scoring and adversarial pattern matching.
- **PII Redaction**: Regex and entity detectors identifying Social Security Numbers, credit cards, emails, and API keys with preview masking (`[REDACTED_EMAIL]`).
- **Keyword & Regex Blocklists**: Validates custom enterprise corporate compliance lists.
- **Verdict Breakdown**: Displays pass/block disposition, matched rule IDs, execution latency overhead, and sanitized prompt payloads.
### View 7: Upstream Credentials Vault (`CredentialsView.tsx`)
Secure management of upstream provider API keys and hyperscaler credentials:
- **AES-256-GCM Vault Management**: Providers credentials (`OpenAI`, `Anthropic`, `Google Gemini`, `AWS Bedrock`, `Google Vertex`, `Azure OpenAI`, `Cohere`, `Mistral`) are stored encrypted at rest.
- **Zero-Plaintext Ingress**: Displays masked key previews (e.g. `sk-ant-api03-...ABCD`) without exposing raw secret material to the browser.
- **Active Deployment Association**: Maps each stored credential to its associated upstream model deployments and routing pools.
- **Credential Health Indicators**: Displays active probe status and expiration warnings for cloud hyperscaler service account tokens.
### View 8: Agentic Hub Explorer (`AgenticView.tsx`)
Operational control center for Model Context Protocol (MCP) and Agent-to-Agent (A2A) communications:
- **MCP Server Registry**: Inspects all registered Streamable HTTP, SSE, and Stdio MCP servers, connection statuses, and transport types.
- **Progressive Tool Discovery**: Explores discovered tool definitions, opaque 256-bit handles (`tool_hdl_...`), and input schemas.
- **Curated Toolsets**: Inspects custom tool groupings (e.g., `git-tools`, `analytics-tools`) mapped to tenant teams.
- **A2A Agent Network**: Discovers registered Linux Foundation A2A v1.0 agents, cryptographic Agent Cards (`.well-known/agent-card.json`), JWS signing status, and active task state streams.
---
## 4. Local Development Workflow
When contributing to or customizing the Operator UI, Vite runs in development mode with hot module replacement (HMR) and proxies backend API calls to a running Go gateway:
```bash
# Terminal 1: Run the Go backend
go run ./cmd/gateway -config configs/config.yaml
# Terminal 2: Run Vite Dev Server with HMR
make ui-dev
```
Vite listens on `http://localhost:5173/` and proxies:
- `/v1/*` $\rightarrow$ `http://localhost:8080/v1/*`
- `/admin/v1/*` $\rightarrow$ `http://localhost:8080/admin/v1/*`
- `/mcp/*` and `/a2a/*` $\rightarrow$ `http://localhost:8080/*`
### Synchronizing Distribution Files
Before committing changes to `web/src/`:
```bash
# 1. Compile TypeScript, build production bundle, and verify budget
make ui-build
# 2. Confirm internal/ui/dist is in sync with web/dist
make ui-check-dist
# 3. Execute UI unit and integration tests
go test -v ./internal/ui/...
```
---
## Observability, Metrics & Health Probes
URL: https://goroute-ai.github.io/docs/deploy/observability
Markdown: https://goroute-ai.github.io/docs/deploy/observability.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Observability, Metrics & Health Probes
Observability helps you answer practical questions: did a request reach a provider, how long did it take, and is the service healthy? GoRoute can expose metrics, pass distributed trace context, and report health information for operators and monitoring tools.
```mermaid
flowchart LR
app["Application"] --> gateway["GoRoute"] --> provider["AI provider"]
gateway --> traces["Request traces"]
gateway --> metrics["Service metrics"]
gateway --> health["Health checks"]
classDef appStyle fill:#153451,stroke:#60A5FA,color:#EFF6FF
classDef gatewayStyle fill:#0B5668,stroke:#22D3EE,color:#ECFEFF
classDef serviceStyle fill:#382F6B,stroke:#A78BFA,color:#F5F3FF
classDef signalStyle fill:#654717,stroke:#FBBF24,color:#FFFBEB
class app appStyle
class gateway gatewayStyle
class provider serviceStyle
class traces signalStyle
class metrics signalStyle
class health signalStyle
```
The detailed setup includes:
1. **Prometheus Metrics Engine (`GET /metrics`)**: In-process Vitter Algorithm R reservoir sampling (`1,024` samples) of latencies, Time-To-First-Token (TTFT), routing overheads, cache-hit ratios, and provider spend.
2. **OTLP/HTTP & W3C Distributed Tracing**: Exports `gen_ai.client` spans (`gen_ai.operation.name`, `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.*`) via `tracing.otlp_endpoint` through the outbound SSRF policy, alongside `traceparent` and `X-Trace-Id` propagation.
3. **Multi-Tier Health Probes & Policy Simulation**: Kubernetes liveness (`/healthz`), readiness (`/readyz`), active upstream deployment probing (`/admin/v1/health/*`), and the stateless Guardrails Policy Simulator (`POST /admin/v1/organizations/{orgID}/guardrails/simulate`).
4. **Embedded Operator Console & Secret-Scrubbed Logs**: Single-binary React UI served at `/ui/` and structured `slog` logs scrubbed via `internal/logger/scrub.go` (`ScrubSecrets`).
See also: [OpenTelemetry & Prometheus Integration](https://goroute-ai.github.io/docs/integrations/otel-and-prometheus), [Embedded Operator Web UI](https://goroute-ai.github.io/docs/deploy/operator-ui), and [Troubleshooting](https://goroute-ai.github.io/docs/deploy/troubleshooting).
```mermaid
flowchart TD
subgraph K8s["Orchestration & Scrapers"]
Prom["Prometheus Scraper"]
K8sProbe["K8s Kubelet Probes"]
Telemetry["OpenTelemetry Collector"]
end
subgraph Gateway["GoRoute AI Gateway"]
MetricsEndpoint["GET /metrics
(Text Format 0.0.4)"]
Liveness["GET /healthz"]
Readiness["GET /readyz"]
AdminHealth["GET /admin/v1/health/deployments
POST /admin/v1/health/probe"]
TraceMiddleware["W3C Trace Middleware
(00-{trace}-{span}-01)"]
end
subgraph Upstream["Upstream Providers"]
OpenAI["OpenAI API"]
Anthropic["Anthropic API"]
end
Prom -->|Scrape 15s| MetricsEndpoint
K8sProbe -->|Liveness Check| Liveness
K8sProbe -->|Readiness Check| Readiness
Telemetry -->|Trace Logs| TraceMiddleware
TraceMiddleware -->|Outbound Traceparent| OpenAI
TraceMiddleware -->|Outbound Traceparent| Anthropic
```
---
## 1. Prometheus Metrics Scraper (`GET /metrics`)
The gateway exposes a protected Prometheus scraper endpoint at `GET /metrics` formatted according to Prometheus exposition specification 0.0.4. Requests require the Master Admin Key via `Authorization: Bearer `.
Metrics are maintained in-memory using Vitter Algorithm R uniform reservoir sampling (`1,024` samples) and omit per-deployment labels to keep metric cardinality bounded.
### Exported Metrics Catalog
| Metric Name | Type | Labels | Description |
| :--- | :--- | :--- | :--- |
| `gateway_requests_total` | Counter | None | Total inference requests processed across all endpoints. |
| `gateway_errors_total` | Counter | None | Total failed inference requests (HTTP 4xx/5xx). |
| `gateway_latency_ms` | Gauge | `quantile="0.5"`, `"0.95"`, `"0.99"` | End-to-end request duration percentiles in milliseconds. |
| `gateway_ttft_ms` | Gauge | `quantile="0.95"` | P95 Time-To-First-Token for streaming responses in milliseconds. |
| `gateway_routing_overhead_micros` | Gauge | `quantile="0.95"` | P95 Stage 1 (heuristic) + Stage 2 (load balancer) routing decision time in microseconds. |
| `gateway_fallbacks_total` | Counter | None | Total failover executions to secondary or tertiary model groups. |
| `gateway_retries_total` | Counter | None | Total intra-group exponential retry attempts. |
| `gateway_cache_hits_total` | Counter | None | Total exact-match responses served directly from Redis/L1 cache. |
| `gateway_ratelimit_rejections_total` | Counter | None | Total requests rejected by RPM/TPM rate limits or financial spend ceilings. |
| `gateway_guardrail_blocks_total` | Counter | None | Total requests intercepted and blocked by security guardrail rules. |
| `gateway_guardrail_evaluations_total` | Counter | `guardrail`, `hook`, `decision` | Total guardrail evaluations segmented by guardrail name, execution hook, and decision (`allow`, `block`, `flag`, `modify`, `error`). |
| `gateway_guardrail_errors_total` | Counter | `guardrail`, `hook` | Total guardrail evaluation errors segmented by guardrail name and hook. |
| `gateway_guardrail_duration_ms` | Histogram | `guardrail`, `hook` | Guardrail evaluation latency histogram in milliseconds (`le` buckets and `+Inf`). |
| `gateway_provider_requests_total` | Counter | `provider` | Request volume segmented by upstream provider (`openai`, `anthropic`, etc.). |
| `gateway_provider_errors_total` | Counter | `provider` | Upstream failure count segmented by provider. |
| `gateway_provider_latency_p95_ms` | Gauge | `provider` | P95 upstream round-trip latency segmented by provider. |
| `gateway_provider_spend_micros_total`| Counter | `provider` | Cumulative spend in integer micro-dollars (10^-6 USD) per provider. |
### Optional OTLP/HTTP Tracing & Structured Log Scrubbing
- **OTLP Distributed Tracing**: Configure `tracing.otlp_endpoint` (and optional `tracing.headers`) to export OpenTelemetry spans with `gen_ai.*` attributes through the outbound SSRF network policy.
- **Structured Log Scrubbing**: `internal/logger/scrub.go` (`ScrubSecrets`) masks API keys, bearer tokens, and credential patterns across structured `slog` fields before logs are emitted.
### Sample Scraper Output
```text
# HELP gateway_requests_total Total inference requests processed by the AI Gateway.
# TYPE gateway_requests_total counter
gateway_requests_total 42180
# HELP gateway_errors_total Total failed requests.
# TYPE gateway_errors_total counter
gateway_errors_total 12
# HELP gateway_latency_ms Request latency percentiles in milliseconds.
# TYPE gateway_latency_ms gauge
gateway_latency_ms{quantile="0.5"} 342
gateway_latency_ms{quantile="0.95"} 890
gateway_latency_ms{quantile="0.99"} 1420
# HELP gateway_ttft_ms Time-To-First-Token in milliseconds.
# TYPE gateway_ttft_ms gauge
gateway_ttft_ms{quantile="0.95"} 184
# HELP gateway_routing_overhead_micros Stage 1 + Stage 2 routing decision overhead in microseconds.
# TYPE gateway_routing_overhead_micros gauge
gateway_routing_overhead_micros{quantile="0.95"} 42
# HELP gateway_cache_hits_total Total exact-match response cache hits.
# TYPE gateway_cache_hits_total counter
gateway_cache_hits_total 8520
# HELP gateway_provider_requests_total Total requests by provider.
# TYPE gateway_provider_requests_total counter
gateway_provider_requests_total{provider="anthropic"} 21040
gateway_provider_requests_total{provider="gemini"} 15300
gateway_provider_requests_total{provider="openai"} 5840
```
### Scraping via Prometheus (`prometheus.yml`)
Add the following scrape configuration to your Prometheus server:
```yaml
scrape_configs:
- job_name: "goroute-ai-gateway"
scrape_interval: 15s
static_configs:
- targets: ["gateway-service.default.svc.cluster.local:8080"]
metrics_path: "/metrics"
```
---
## 2. W3C Distributed Tracing & Header Propagation
To ensure transparent traceability across distributed microservice topologies, the gateway natively implements the **W3C Trace Context** specification (`traceparent`).
### Trace Identifier Generation (`internal/observability/tracing.go`)
Every incoming HTTP request is evaluated by `TraceMiddleware`:
1. **Trace ID Ingestion**: Checks for an existing W3C `traceparent` header (format: `00-{trace_id}-{span_id}-{flags}`). If absent, checks `X-Trace-Id`. If neither is provided, generates a fresh 128-bit random Trace ID (32 hex characters).
2. **Span ID Generation**: Generates a new 64-bit random Span ID (16 hex characters) representing the gateway's processing span.
3. **Outbound Header Injection**: Automatically injects `traceparent` and `X-Trace-Id` into:
- Client response headers.
- Outbound HTTP requests sent to upstream cloud providers (OpenAI, Anthropic, Bedrock, Vertex).
- Webhook push notifications (A2A transactional outbox).
```text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
X-Trace-Id: 4bf92f3577b34da6a3ce929d0e0e4736
X-Gateway-Request-Id: req_8f12a39b4c0e
```
### Upstream Attribution Telemetry Headers
Every data plane response includes standard gateway telemetry headers:
| Header Name | Value Format | Description |
| :--- | :--- | :--- |
| `X-Trace-Id` | `32-char hex` | W3C 128-bit distributed trace correlation ID. |
| `X-Gateway-Request-Id` | `req_` | Unique per-request gateway tracking ID. |
| `X-Gateway-Provider` | `string` | Resolved upstream provider (e.g. `anthropic`, `gemini`). |
| `X-Gateway-Model` | `string` | Physical upstream model invoked (e.g. `claude-3-5-sonnet-20241022`). |
| `X-Gateway-Cost-Micros`| `integer` | Micro-dollars (10^-6 USD) charged for the operation. |
| `X-Gateway-Routing-Time`| `duration` | Microseconds elapsed making Stage 1 + 2 routing decisions. |
| `X-Gateway-Cache` | `HIT` / `MISS` | Indicates whether the response was served from cache. |
---
## 3. Deployment Health Probes
The gateway provides three distinct health check interfaces designed for load balancers, orchestrators, and automated site reliability tooling.
### Liveness Probe (`GET /healthz`)
A lightweight probe returning HTTP 200 as long as the Go HTTP process is running:
```bash
curl -i http://localhost:8080/healthz
```
Response:
```http
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
X-Trace-Id: c8901f43a0e14cb789123456789abcde
OK
```
### Readiness Probe (`GET /readyz`)
Verifies that all configured stateful storage dependencies are operational. When running in `postgres` mode, the probe executes an active database ping and verifies Redis connectivity.
```bash
curl -i http://localhost:8080/readyz
```
- If both PostgreSQL and Redis respond: returns `HTTP 200 OK`.
- If either dependency is unreachable or times out: returns `HTTP 503 Service Unavailable`.
### Active Deployment Health Inspector (`GET /admin/v1/health/deployments`)
Provides the active health status across all configured upstream model deployments:
```bash
curl -X GET "http://localhost:8080/admin/v1/health/deployments" \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me"
```
Response:
```json
{
"data": [
{
"deployment_id": "dep_openai_gpt4o",
"deployment_name": "OpenAI US-East Production",
"provider": "openai",
"model": "gpt-4o",
"status": "healthy",
"consecutive_failures": 0,
"probe_latency_ms": 142,
"last_checked_at": "2026-11-11T16:00:00Z"
},
{
"deployment_id": "dep_anthropic_sonnet",
"deployment_name": "Anthropic Direct",
"provider": "anthropic",
"model": "claude-3-5-sonnet",
"status": "degraded",
"consecutive_failures": 1,
"probe_latency_ms": 1250,
"last_error": "rate_limit: 429 Too Many Requests",
"last_checked_at": "2026-11-11T16:00:00Z"
}
]
}
```
### Triggering Instant Active Sweep (`POST /admin/v1/health/probe`)
Operators can force an immediate, non-blocking health probe sweep across all registered deployments (testing credentials and upstream connectivity via `ListModels` or ping):
```bash
curl -X POST "http://localhost:8080/admin/v1/health/probe" \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me"
```
The gateway audits the trigger action in `audit_logs` under action `health.deployments.probed` and updates all deployment circuit breaker states accordingly.
---
## 4. Kubernetes Production Manifest Example
Configure your Kubernetes deployment pod spec with tailored probes:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: goroute-ai-gateway
spec:
replicas: 3
template:
spec:
containers:
- name: gateway
image: ghcr.io/goroute-ai/ai-gateway-go:v1.8.0
ports:
- containerPort: 8080
name: http
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 2
```
---
## Production Readiness Checklist
URL: https://goroute-ai.github.io/docs/deploy/production-checklist
Markdown: https://goroute-ai.github.io/docs/deploy/production-checklist.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Production Readiness Checklist
Use this guide before making a GoRoute deployment available to your team or customers. It walks through the configuration, access controls, network boundaries, and service checks that operators should review. Follow the linked guides for details about each setting.
Before promoting GoRoute AI Gateway to production traffic, verify each control below against your environment variables, `config.yaml`, and Kubernetes or container deployment manifests.
---
## 1. Environment & Secret Validation
Setting `ENV=production` or `GATEWAY_ENV=production` (or `server.environment: production`) activates strict startup validation in `Config.Validate()`:
- [ ] **Enable production mode**: Set `GATEWAY_ENV=production`.
- [ ] **Replace default Master Admin Key**: Set `GATEWAY_MASTER_ADMIN_KEY` to a high-entropy secret (must not equal `sk-gw-master-admin-default-key-change-me`).
- [ ] **Replace default Encryption Key**: Set `GATEWAY_MASTER_ENCRYPTION_KEY` to a unique 32-byte hex (64 hex characters) AES-256-GCM Key Encryption Key (must not equal `0123456789abcdef...`).
- [ ] **Replace default JWT Secret**: Set `GATEWAY_JWT_SECRET` to a strong HMAC secret (must not equal `gateway-jwt-hmac-secret-change-me`), or configure OIDC JWKS verification.
- [ ] **Enable PostgreSQL persistence**: Configure `GATEWAY_POSTGRES_DSN` with TLS (`sslmode=require` or `sslmode=verify-full`) so `storage.mode` resolves to `postgres` rather than ephemeral `memory`.
---
## 2. Outbound SSRF Firewall & Network Policy
GoRoute routes all provider, MCP, A2A, and OTLP HTTP egress through `internal/security/network`:
- [ ] **Block private networks**: Verify `security.network.outbound_ssrf.allow_private_networks` is `false` (default) and `GATEWAY_SECURITY_SSRF_ALLOW_PRIVATE` is unset or `false`.
- [ ] **Require HTTPS upstreams**: Verify `security.network.outbound_ssrf.allow_insecure_http` is `false` (default) and `GATEWAY_SECURITY_SSRF_ALLOW_INSECURE_HTTP` is unset or `false`.
- [ ] **Explicit CIDR allowlist (if needed)**: If routing to private VPC inference endpoints (e.g., internal vLLM), use `security.network.outbound_ssrf.allowed_private_cidrs` rather than opening all private networks.
- [ ] **Block cloud metadata IPs**: Ensure `169.254.169.254` remains blocked via `security.network.outbound_ssrf.blocked_ips`.
- [ ] **Restrict ingress callers**: Set `security.network.inbound_caller.allowed_client_cidrs` if the gateway sits behind a known load balancer or API gateway CIDR.
---
## 3. Operator RBAC, OIDC & HTTP Security Headers
- [ ] **Disable master-key data-plane fallback**: Confirm `auth.allow_master_key_data_plane` (`GATEWAY_ALLOW_MASTER_KEY_DATA_PLANE`) is `false` (default). Data-plane inference calls must authenticate with tenant Virtual Keys (`Authorization: Bearer sk-gw-...`).
- [ ] **Configure Inbound OIDC / JWT**: Set `auth.oidc_jwks_url` (`GATEWAY_OIDC_JWKS_URL`), `auth.jwt_issuer` (`GATEWAY_JWT_ISSUER`), and `auth.jwt_audience` (`GATEWAY_JWT_AUDIENCE`) for operator control-plane SSO.
- [ ] **Verify RBAC roles**: Assign least-privilege operator roles (`platform_admin`, `org_admin`, `org_viewer`) to users; `org_admin` and `org_viewer` are strictly scoped to their organization.
- [ ] **Enable HSTS & CORS allowlists**: Set `server.enable_hsts: true` when terminating TLS at the edge, and restrict `server.cors_allowed_origins` to your Operator UI origin instead of `*`.
- [ ] **Disable MCP stdio in production**: Keep `mcp.allow_stdio_servers: false` (enforced fail-closed by `Config.Validate()`).
---
## 4. Fail-Closed Rate Limits & Budget Enforcement
- [ ] **Enable Redis-backed rate limiting**: Confirm `rate_limit.enabled: true` (default) with `rate_limit.redis_timeout` tuned (default `50ms`) and Redis Cluster or Sentinel configured via `GATEWAY_REDIS_ADDRS`.
- [ ] **Confirm fail-closed behavior**: When Redis is unreachable or team lookups fail, GoRoute fails closed (`HTTP 429` / `503`) rather than allowing unmetered traffic.
- [ ] **Monitor unpriced usage**: Audit `unpriced_requests` and `unpriced_tokens` in FinOps spend summaries; inferred catalog rates are never treated as verified for budgeted traffic.
---
## 5. Circuit Breakers & Reliability
- [ ] **Configure shared Redis circuit breakers**: In multi-replica deployments, ensure all pods share the same Redis cluster/instance so circuit-breaker failure counts and half-open probe leases synchronize across replicas.
- [ ] **Tune breaker thresholds**: Review `router.failure_threshold` (default `3`), `router.failure_window` (default `30s`), and `router.default_cooldown` (default `60s`, or `GATEWAY_ROUTER_DEFAULT_COOLDOWN`).
- [ ] **Verify retry & fallback bounds**: Confirm `router.default_max_retries` (default `2`), `router.base_retry_delay` (`50ms`), and `router.max_retry_delay` (`2s`) match your upstream SLA budgets.
---
## 6. File Storage Quotas & Upload Concurrency
- [ ] **Enforce per-org storage quota**: Verify `server.max_file_storage_bytes_per_org` (`GATEWAY_MAX_FILE_STORAGE_BYTES_PER_ORG`, default `5 GiB`) matches your disk capacity. Requests exceeding quota return `413 storage_quota_exceeded`.
- [ ] **Bound concurrent uploads**: Keep `server.max_concurrent_file_uploads` (`GATEWAY_MAX_CONCURRENT_FILE_UPLOADS`, default `8`) bounded to protect pod memory and disk I/O (`429 upload_concurrency_limit`).
- [ ] **Mount writable temp volume**: If running with `readOnlyRootFilesystem: true` in Kubernetes, mount an `emptyDir` or PVC for multipart upload spill files.
---
## 7. Pre-Flight Load, Security & Canary Verification
Run the built-in verification suites before cutting over production traffic:
```bash
# 1. Containerized security, vulnerability, and k6 load suite
# (gitleaks, trivy, govulncheck, k6, pgbench, redis-benchmark)
make test-prod-containers
# 2. Go load benchmarks & optional 5-minute soak test
GATEWAY_RUN_SOAK=1 make test-load
# 3. Security boundary & auth test suite
make test-security
# 4. Key-gated live provider canary checks (requires provider API keys)
make test-canary
```
:::info
See [Kubernetes & Kustomize Deployment](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize) for container hardening and health probe configuration, and [Observability, Metrics & Health Probes](https://goroute-ai.github.io/docs/deploy/observability) for Prometheus alerting thresholds.
:::
---
## Troubleshooting & Recovery Runbooks
URL: https://goroute-ai.github.io/docs/deploy/troubleshooting
Markdown: https://goroute-ai.github.io/docs/deploy/troubleshooting.md
Section: Control Plane Documentation > Deploy and operate GoRoute
# Troubleshooting & Recovery Runbooks
This guide provides root-cause diagnostic workflows and operational recovery runbooks for common production failure modes in GoRoute AI Gateway.
```mermaid
flowchart TD
Error["Client Encounters Error Response"] --> CheckCode{"Check HTTP Status & Error Code"}
CheckCode -->|"HTTP 402"| Budget["Budget Exceeded (budget_exceeded_error)
→ Runbook 1: Adjust Spend Limits"]
CheckCode -->|"HTTP 429"| RateLimit["Rate Limited (rate_limit_error)
→ Runbook 2: Sliding-Window Quota / Leases"]
CheckCode -->|"HTTP 403 / 502 SSRF"| NetworkBlock["SSRF or Private Network Block
→ Runbook 3: Network Policy & CIDRs"]
CheckCode -->|"HTTP 502 / 503"| CircuitBreaker["Circuit Breaker Tripped (OPEN)
→ Runbook 4: Cooldown & Probing"]
CheckCode -->|"HTTP 401 / 403 Auth"| RBAC["Auth / Permission Error
→ Runbook 5: 4-Layer RBAC & Keys"]
```
---
## 1. Unified Error Taxonomy Reference
All data plane errors conform to standard OpenAI error formats with machine-readable `type` and `code` properties:
```json
{
"error": {
"type": "rate_limit_error",
"code": "virtual_key_rpm_limit_exceeded",
"message": "Virtual key sk-gw-a1b2 has exceeded its limit of 100 requests per minute."
}
}
```
| HTTP Status | Error `type` | Typical Error `code` | Root Cause |
| :--- | :--- | :--- | :--- |
| **`400 Bad Request`** | `invalid_request_error` | `invalid_json`, `missing_model` | Request body syntax error or unrecognized parameters. |
| **`401 Unauthorized`** | `authentication_error` | `invalid_api_key`, `key_expired` | Missing, malformed, or expired client virtual key. |
| **`402 Payment Required`** | `budget_exceeded_error`| `org_budget_exceeded`, `key_budget_exceeded` | Hard micro-dollar spending limit reached by tenant or key. |
| **`403 Forbidden`** | `permission_error` | `model_forbidden`, `mcp_forbidden` | 4-layer RBAC policy prevents key from accessing target model or server. |
| **`404 Not Found`** | `not_found_error` | `model_not_found`, `task_not_found` | Requested virtual alias or A2A task does not exist. |
| **`429 Too Many Requests`**| `rate_limit_error` | `rate_limit_rpm_exceeded`, `concurrency_limit` | Sliding-window RPM/TPM limit or concurrent in-flight lease limit reached. |
| **`502 Bad Gateway`** | `upstream_provider_error`| `upstream_auth_error`, `upstream_bad_gateway` | Upstream provider rejected credential or failed with 5xx. |
| **`503 Service Unavailable`**| `upstream_provider_error`| `circuit_open`, `all_deployments_down` | Circuit breaker tripped; all primary and fallback deployments offline. |
| **`504 Gateway Timeout`** | `upstream_provider_error`| `upstream_timeout` | Upstream provider exceeded write/read timeout. |
---
## 2. Runbook 1: Financial Spend Budget Exceeded (`HTTP 402`)
### Symptom
Client receives HTTP 402 with error type `budget_exceeded_error`:
```json
{
"error": {
"type": "budget_exceeded_error",
"code": "org_budget_exceeded",
"message": "Configured financial spend budget exceeded (org_budget_exceeded)"
}
}
```
### Diagnosis
1. Inspect the caller's virtual key and tenant association:
```bash
curl -X GET "http://localhost:8080/admin/v1/virtual-keys?prefix=sk-gw-a1b2" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
2. Check current accumulated spend vs maximum budget in PostgreSQL:
```sql
SELECT id, name, spent_micros, max_budget_micros,
(spent_micros::float / NULLIF(max_budget_micros, 0) * 100) AS pct_used
FROM organizations
WHERE slug = 'acme-corp';
```
### Remediation
- **Option A: Increase Organization or Key Budget via Admin API**:
```bash
curl -X PATCH "http://localhost:8080/admin/v1/organizations/org_12345" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{"max_budget_micros": 500000000}' # $500.00 USD
```
- **Option B: Reset Accumulated Spend for a New Billing Cycle**:
```sql
UPDATE organizations
SET spent_micros = 0, budget_reset_at = NOW()
WHERE id = 'org_12345';
```
The gateway will automatically clear the Redis spend accumulator key (`gw:spend:org:org_12345`).
---
## 3. Runbook 2: Rate Limit or Concurrency Rejection (`HTTP 429`)
### Symptom
Requests fail with HTTP 429. The response contains a `Retry-After: ` header.
### Diagnosis
1. Check gateway logs for rate limiter evaluation messages:
```bash
grep "rate_limit_exceeded" /var/log/goroute/gateway.log
```
2. Check Redis sliding-window counters:
```bash
redis-cli KEYS "gw:rl:*:sk-gw-a1b2:*"
redis-cli GET "gw:rl:conc:sk-gw-a1b2"
```
### Remediation
- **If Requests Per Minute (RPM) or Tokens Per Minute (TPM) are exhausted**:
Update the virtual key limits:
```bash
curl -X PATCH "http://localhost:8080/admin/v1/virtual-keys/key_12345" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{"rpm_limit": 500, "tpm_limit": 100000}'
```
- **If Concurrency Locks are Stale**:
In rare scenarios where client connections abort without cleanly releasing concurrent lease counters, expire the concurrency key:
```bash
redis-cli DEL "gw:rl:conc:key_12345"
```
---
## 4. Runbook 3: Outbound SSRF & Private Network Blocks
### Symptom
Requests to self-hosted models (vLLM, Ollama, TGI) or private VPC endpoints fail with errors such as:
```text
[502:upstream_error] network: private/local network address is forbidden by SSRF policy
```
Or:
```text
network: destination IP is in the forbidden list
```
### Root Cause
The gateway enforces a zero-trust outbound security policy (`internal/security/network/policy.go`):
- Prohibits loopback (`127.0.0.0/8`, `::1/128`).
- Prohibits RFC 1918 private subnets (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`).
- Prohibits cloud metadata endpoints (`169.254.169.254`, `fd00:ec2::254`).
- Blocks plaintext `http://` schemes when `allow_insecure_http` is `false`.
### Remediation for Internal Models
If you are hosting private inference engines within a secure VPC, configure explicit CIDR allowlists or enable private network routing in `configs/config.yaml`:
```yaml
security:
network:
outbound_ssrf:
allow_private_networks: true
allowed_schemes:
- "https"
- "http"
allowed_private_cidrs:
- "10.240.0.0/16" # Your internal Kubernetes pod CIDR
- "192.168.1.0/24" # On-premise GPU cluster
```
> [!CAUTION]
> Never configure `169.254.169.254` or `0.0.0.0/0` in `allowed_private_cidrs`. Cloud instance metadata must remain permanently blocked to prevent credential theft.
---
## 5. Runbook 4: Tripped Circuit Breakers (`CircuitOpen`)
### Symptom
Inference requests fail immediately with HTTP 503 without reaching the upstream provider:
```json
{
"error": {
"type": "upstream_provider_error",
"code": "circuit_open",
"message": "Upstream deployment dep_openai_east is isolated in circuit breaker cooldown"
}
}
```
### Diagnosis
1. Inspect deployment statuses:
```bash
curl -X GET "http://localhost:8080/admin/v1/health/deployments" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
2. Verify if the deployment is in cooldown in Redis:
```bash
redis-cli GET "gw:router:cooldown:dep_openai_east"
```
### Remediation
- **Wait for Cooldown Expiration**: By default, the deployment remains isolated for `router.default_cooldown` (60 seconds). Once expired, the circuit enters `HALF_OPEN` state, allowing a single trial probe to verify recovery.
- **Force Immediate Recovery via Active Probe**:
```bash
curl -X POST "http://localhost:8080/admin/v1/health/probe" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
- **Manual Reset**: Delete the Redis cooldown lock directly:
```bash
redis-cli DEL "gw:router:cooldown:dep_openai_east"
```
---
## 6. Runbook 5: 4-Layer RBAC Authorization Failures (`HTTP 403`)
### Symptom
Client receives HTTP 403:
```json
{
"error": {
"type": "permission_error",
"code": "model_forbidden",
"message": "Virtual key sk-gw-a1b2 is not authorized to use model \"o3-mini\""
}
}
```
### Diagnosis: Trace the 4-Layer Hierarchy
The gateway checks 4 hierarchical layers in order. If any layer denies access, the request is rejected fail-closed:
```mermaid
flowchart TD
Req["Request for 'o3-mini'"] --> L1{"Layer 1: Global Catalogue
Is model active?"}
L1 -->|Yes| L2{"Layer 2: Org Subscription
Is provider enabled for org?"}
L1 -->|No| Block["403 Forbidden"]
L2 -->|Yes| L3{"Layer 3: Org Model Policy
Is model approved?"}
L2 -->|No| Block
L3 -->|Yes| L4{"Layer 4: Virtual Key Allowlist
Is model in allowed_models?"}
L3 -->|No| Block
L4 -->|Yes| Allow["200 OK (Allowed)"]
L4 -->|No| Block
```
### Remediation Queries
1. **Verify Layer 2 Subscription**:
```sql
SELECT * FROM org_provider_subscriptions
WHERE org_id = 'org_default' AND provider = 'openai';
```
2. **Verify Layer 3 Model Approval**:
```sql
SELECT * FROM org_model_policies
WHERE org_id = 'org_default' AND global_model_id = 'openai/o3-mini';
```
3. **Verify Layer 4 Key Allowlists**:
```sql
SELECT allowed_models FROM virtual_keys WHERE key_prefix = 'sk-gw-a1b2';
```
Update the key allowlist to include the requested model or `"*"`.
---
## Engineering Guide for Coding Agents
URL: https://goroute-ai.github.io/docs/contributing/claude-code-and-agents
Markdown: https://goroute-ai.github.io/docs/contributing/claude-code-and-agents.md
Section: Control Plane Documentation > Contribute
# Engineering Guide for Coding Agents
> [!NOTE]
> This guide is for **contributors developing the `ai-gateway-go` repository itself**. If you want to route **Claude Code** or IDE traffic *through* a running GoRoute AI Gateway instance, see [Claude Code Integration](https://goroute-ai.github.io/docs/integrations/claude-code) and [Cursor, Continue & Cline](https://goroute-ai.github.io/docs/integrations/cursor-and-ides).
GoRoute AI Gateway (`goroute-ai/ai-gateway-go`) is engineered from the ground up to be maintainable, verifiable, and extensible by both human software engineers and autonomous AI coding agents (such as **Claude Code**, **Cursor**, **GitHub Copilot**, and **Google Antigravity**).
To guarantee that agent-generated pull requests maintain architectural invariants, code quality, and security boundaries, the codebase incorporates a hierarchical context architecture consisting of:
1. **Root `CLAUDE.md`**: Compact project instructions and essential commands (under 100 lines).
2. **Modular `.claude/rules/`**: Scoped sub-domain guidelines (`go-backend.md`, `frontend-web.md`, `api-contracts.md`).
3. **Universal `AGENTS.md`**: Portable repository guidance for all autonomous agent engines.
4. **Deterministic Verification Loops**: Non-interactive commands to validate syntax, typing, static analysis, race conditions, contracts, and frontend bundle budgets.
---
## 1. Root `CLAUDE.md` Architectural Invariants
The root `CLAUDE.md` file serves as the immediate instruction entrypoint for Claude Code sessions. To ensure high token efficiency, it is maintained under a strict 100-line limit while establishing the 7 core repository invariants:
```mermaid
graph TD
Root["CLAUDE.md: Under 100 Lines
(Immediate Context)"]
Root --> Inv1["1. Micro-Dollar FinOps (Integer Arithmetic)"]
Root --> Inv2["2. SSRF & Egress Protection (internal/security/network)"]
Root --> Inv3["3. Fail-Closed 4-Layer RBAC (Org/Team/User/Key)"]
Root --> Inv4["4. AES-256-GCM Credential Vault"]
Root --> Inv5["5. Contract-Guided Specs (OpenAPI 3.1 & Protos)"]
Root --> Inv6["6. Single-Binary Embedded UI (Under 400 KB Budget)"]
Root --> Inv7["7. Immutable Model Pricing Audit History"]
```
### The 7 Non-Negotiable Invariants
1. **Integer Micro-Dollar FinOps**: All financial and spend calculations must use 64-bit integer micro-dollars (1 USD = 1,000,000 micro-dollars) with intermediate pico-dollar precision. Never use floating-point types (`float32` / `float64`) for money.
2. **SSRF & Outbound Egress Protection**: Outbound network requests must use `internal/security/network` safe HTTP transport (`SafeTransport`, `DialContext`). Never use raw unvalidated `http.DefaultClient` or unpinned IPs for provider egress.
3. **Fail-Closed RBAC & Auth**: Multi-tenant authorization (`internal/auth/rbac.go`) is strictly fail-closed across all 4 tiers (Organization → Team → User → Virtual Key).
4. **Credential Vault**: Upstream provider API keys and secrets must remain encrypted via AES-256-GCM (`internal/storage/vault.go`) at rest and in transit.
5. **Contract-Guided Architecture**: API and wire contracts are defined in `docs/openapi/gateway.openapi.yaml`, `docs/specs/mcp/`, and `docs/specs/a2a/`. Verify contracts with `go run ./cmd/specgen`.
6. **Single-Binary Web UI Embedding**: The Operator UI in `web/` compiles into `internal/ui/dist/` and is embedded via Go `embed.FS` (`//go:embed all:dist`). Total frontend bundle size must strictly respect budget constraints (under 400 KB total; verified via `web/scripts/check-budget.mjs`).
7. **Immutable Pricing History**: Model pricing updates must append an immutable audit record to `model_pricing_history`.
---
## 2. Modular Rules: `.claude/rules/`
Rather than crowding the root prompt with domain-specific rules, the repository utilizes modular rule definitions located in `.claude/rules/`. AI agents activate these instructions dynamically based on the files they touch.
### `go-backend.md`: Pure Go Backend Architecture
Located at `.claude/rules/go-backend.md`, this guide enforces Go idioms and safety standards:
- **Strict Dependency Direction**: Package dependencies point strictly inward toward domain definitions in `internal/domain`:
- `cmd/`: Application entrypoints (`cmd/gateway`, `cmd/specgen`, `cmd/catalog-sync`).
- `pkg/`: Public, stable, embeddable SDKs (`pkg/gateway`).
- `internal/`: Private implementation packages (`internal/server`, `internal/router`, `internal/adapter`, etc.).
- `internal/domain/`: Foundational domain types, interfaces, request/response models, and error codes.
- **Chi Router Conventions**:
- Handlers follow standard `http.HandlerFunc` signatures: `func (s *GatewayServer) handleX(w http.ResponseWriter, r *http.Request)`.
- Path parameters extracted via `chi.URLParam(r, "paramName")`.
- Query parameters extracted via `r.URL.Query().Get("key")`.
- JSON serialization via `writeJSON(w, statusCode, data)`.
- Unified error serialization via `domain.WriteHTTPError(w, err)`.
- **Zero-Allocation Hot Path**: Hot inference routes (`POST /v1/chat/completions`) must avoid unnecessary heap allocations, redundant JSON reflection, or blocking synchronous database writes.
- **Safe Network Egress**: Direct instantiation of `&http.Client{}` is prohibited. All outbound requests must use `network.NewSafeTransport` with DNS pinning, private IP filtering, and link-local address blocking.
### `frontend-web.md`: Operator Web UI Invariants
Located at `.claude/rules/frontend-web.md`, this guide establishes rules for modifying the Operator Web UI:
- **Technology Stack**: React 19 + TypeScript + Vite + Tailwind CSS v3 (dark mode enabled via slate/zinc palette).
- **Icon Imports**: Use `lucide-react` with tree-shaking named imports (e.g. `import { Activity, Shield } from 'lucide-react'`). Avoid importing the entire library to prevent bundle bloat.
- **Bundle Budget Verification**: Every pull request modifying `web/` must pass `check-budget.mjs` with uncompressed size under 380 KB and gzip under 110 KB.
- **Distribution Synchronization**: When frontend source files are changed, `make ui-build` must be executed to rebuild the bundle and sync artifacts to `internal/ui/dist/` before committing.
### `api-contracts.md`: Protocol & Contract Specifications
Located at `.claude/rules/api-contracts.md`, this rule governs wire formats and backwards compatibility:
- **Authoritative Contract Files**:
- Gateway OpenAPI: `docs/openapi/gateway.openapi.yaml` (OpenAPI 3.1.0).
- MCP OpenAPI & Schema: `docs/openapi/mcp.openapi.yaml` and `docs/specs/mcp/2026-07-28/schema.json`.
- A2A OpenAPI & Protobuf: `docs/openapi/a2a.openapi.yaml` and `docs/specs/a2a/v1.0/a2a.proto`.
- **Validation Utility**: Agents must execute `go run ./cmd/specgen` to ensure that route schemas, protobuf definitions, and documentation match the Go code.
- **Attribution Headers**: Responses must include standard attribution telemetry (`X-Gateway-Request-Id`, `X-Gateway-Provider`, `X-Gateway-Model`, `X-Gateway-Cost-Micros`, `X-Gateway-Routing-Time`, `X-Gateway-Cache`).
---
## 3. Universal Portable Guide: `AGENTS.md`
`AGENTS.md` is the cross-engine specification file designed for tools like Cursor, Copilot Workspace, Antigravity, and autonomous agent loops. It emphasizes:
- **Codebase Mapping**: Directs agents to read `docs/CODEBASE_MAP.md` before exploring code.
- **Change Expectations**: Agents must state the exact commands executed and tests run in their change summaries.
- **Preservation of Unrelated Work**: Agents must not revert, rewrite, or touch unrelated code or user changes.
- **Self-Correction Strategy**: When a test fails or `go vet` produces warnings, agents must inspect the failure, apply the minimal targeted fix, and rerun the test suite rather than disabling tests.
---
## 4. Deterministic Verification Loops
Autonomous agents must execute the deterministic verification pipeline before declaring any task complete. The commands below are fast, reproducible, and runnable locally or in CI:
| Step | Verification Goal | Command | Success Criteria |
| :--- | :--- | :--- | :--- |
| **1. Formatting** | Check Go code formatting | `gofmt -l .` | Output must be empty (0 files listed). |
| **2. Static Analysis** | Go type and idiom validation | `go vet ./...` | Exits with return code 0; zero warnings. |
| **3. API Specs** | Validate OpenAPI & Protobuf contracts | `go run ./cmd/specgen` | Passes contract validation without errors. |
| **4. Unit Tests** | Concurrent race detection | `go test -race -count=1 ./internal/... ./pkg/...` | All package unit tests pass under Go race detector. |
| **5. Focused Test** | Rapid feedback during edits | `go test -v -race -run ./internal//...` | Specified unit test passes cleanly. |
| **6. Frontend Build** | Build UI & verify bundle budget | `make ui-build` | Passes Vite build and `check-budget.mjs` threshold. |
| **7. UI Dist Sync** | Confirm embedded dist is mirrored | `make ui-check-dist` | `git status` shows zero uncommitted diffs in `internal/ui/dist`. |
| **8. Binary Build** | Compile production executable | `go build ./cmd/gateway` | Produces executable binary `gateway`. |
| **9. Full Check** | Comprehensive pre-flight check | `make release-check` | All verification checks pass end-to-end. |
### Example Agent Loop Script
Agents can run this one-line verification recipe to confirm entire repository integrity:
```bash
gofmt -l . && go vet ./... && go test -race -count=1 ./internal/server ./internal/router ./internal/auth && go build ./cmd/gateway
```
> [!TIP]
> When testing data plane changes without running live upstream providers, use the built-in mock provider adapter (`internal/adapter/mock.go`) or the lightweight local test server (`configs/config.yaml`) to achieve 100% deterministic test coverage without network dependencies.
---
## Release Notes
URL: https://goroute-ai.github.io/docs/releases
Markdown: https://goroute-ai.github.io/docs/releases.md
Section: Control Plane Documentation > What changed
# Release Notes
GoRoute AI Gateway follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Each release includes automated SQL schema migrations, OpenAPI contract validation, and multi-replica Kubernetes verification.
---
## Release History
| Version | Release Date | Highlights |
| :--- | :--- | :--- |
| **[v1.8.0](https://goroute-ai.github.io/docs/releases/v1.8.0)** | 2026-11-16 | Two-Layer Guardrails Engine (`19` built-in drivers, `7` hooks, Starlark `custom_code`, `WithGuardrailScanner`), `POST /v1/guardrails/apply`, tool/MCP taint tracking, Light/Dark Operator UI with 4-Tab Guardrails Console, Helm chart (`0.3.0`), and migrations `0016`–`0019`. |
| **[v1.7.0](https://goroute-ai.github.io/docs/releases/v1.7.0)** | 2026-11-14 | SSRF & media fetch hardening, pure-SDK parity (`ConnectRealtime`, `WithHTTPClient`), 63-path `specgen` parity, `round-robin` routing, Helm chart (`0.2.0`), and migrations `0013`–`0015`. |
| **[v1.6.0](https://goroute-ai.github.io/docs/releases/v1.6.0)** | 2026-11-13 | Maximum provider & model compatibility, modality FinOps, `0012` migration, hardened Kustomize manifests (`deployments/k8s/`), and containerized production test suite. |
| **[v1.5.0](https://goroute-ai.github.io/docs/releases/v1.5.0)** | 2026-11-13 | Operator RBAC & inbound JWT/OIDC, Redis Cluster/Sentinel support, durable spend outbox, shared circuit breakers, streaming guardrail redaction, and OTLP tracing. |
---
## Upgrade Guide
### 1. Review Breaking & Security Changes
Before upgrading across minor versions:
- Check the target release notes for any new required environment variables or production validation rules (for example, `v1.5.0` introduced `GATEWAY_ENV=production` default-secret rejection and disabled master-key data-plane fallback).
- Review `configs/config.example.yaml` for newly added configuration blocks.
### 2. Database Migrations
GoRoute embeds all PostgreSQL migrations (`migrations/0001_*.sql` through `0019_*.sql`) directly in the gateway binary and applies pending up-migrations atomically at startup (or via `-migrate-only` / the Helm `migration` pre-upgrade hook job).
For zero-downtime rolling upgrades in Kubernetes:
1. Ensure your PostgreSQL user has DDL permissions (`CREATE TABLE`, `ALTER TABLE`, `CREATE INDEX`) or run a pre-upgrade migration job.
2. Take a PostgreSQL snapshot before applying major schema changes (such as `0008` spend outbox, `0012` cache-write and operations columns, `0013`–`0015` user/team/org concurrency and budget limits, or `0016`–`0019` guardrails registry, tool policies, and usage metrics).
3. Upgrade the gateway image tag and monitor `/readyz` before draining old pods:
```bash
kubectl -n goroute-prod set image deployment/gateway gateway=ghcr.io/goroute-ai/ai-gateway-go:v1.8.0
kubectl -n goroute-prod rollout status deployment/gateway --timeout=120s
```
### 3. Verify Post-Upgrade Health
After rollout completes:
- Confirm `GET /healthz` and `GET /readyz` return `200 OK`.
- Run `make test-canary` against a staging environment to validate live provider routing and modality endpoints.
---
## v1.8.0
URL: https://goroute-ai.github.io/docs/releases/v1.8.0
Markdown: https://goroute-ai.github.io/docs/releases/v1.8.0.md
Section: Control Plane Documentation > What changed
# v1.8.0
**Release Date:** 2026-11-16
**Theme:** Multi-Provider Guardrails Engine, 19 Built-In Drivers, Zero-Trust Tool/MCP Governance & Light Blue Dark/Light Operator Web UI
---
## Two-Layer Guardrails Engine, 19 Built-In Drivers & Tool/MCP Governance
- **Baseline Toxicity Persistence & Named Registry (`migrations/0016`, `0017`)**: Persists `GuardrailPolicy` toxicity filter and 5 content category lists in PostgreSQL alongside tenant-scoped `guardrail_definitions` (Vault AES-256-GCM encrypted secrets and DEK rotation) and hierarchical `guardrail_bindings` (`Org` → `Team` → `VirtualKey` → `EndUser` → `ModelGroup` → `Tag`, with `default_on` and anti-bypass `is_locked` enforcement).
- **Pipeline v2 Lifecycle & Load Balancing**: Supports all 7 execution hooks (`pre_call`, concurrent canceling `during_call`, `post_call`, async `logging_only`, `pre_mcp_call`, `during_mcp_call`, `post_mcp_call`), `stream_scope`, tag-based `mode_rule` routing, weighted `RouterLB` cooldown load balancing, and `goroute_extensions` (`guardrails`, `tags`) across inference, non-chat, and OpenAPI contracts (`73` Unified OpenAPI paths: `24` `/v1` + `49` `/admin/v1`).
- **Extensibility (Public Go SDK, Rich Webhooks & Starlark `custom_code`)**: Adds exported `pkg/gateway.GuardrailScanner` (`WithGuardrailScanner`, `allow`/`block`/`flag`/`modify`), rich `generic_http` webhooks with independent per-scanner timeouts and fail-open handling, and sandboxed Starlark `custom_code` scripts (`apply_guardrail`) with deterministic `50,000`-step limits, context deadlines, zero filesystem/import access, and SSRF-checked `http_get`/`http_post`.
- **19 Built-In Drivers, LLM Judges & Streaming Windowing**: Adds `openai_moderation`, `bedrock` (SigV4), `azure/prompt_shield`, `azure/text_moderations`, `model_armor`, `presidio`, `lakera_v2`, `guardrails_ai`, `pangea`, `local_content_filter`, `llm_as_a_judge`, `decision_model` (with recursion loop protection), cross-chunk `StreamWindowEvaluator` masking, and incremental streamed `delta.tool_calls` scanning.
- **Tool & Zero-Trust MCP Governance (`migrations/0018`)**: Adds rule-based `tool_permission`, persistent cross-turn `tool_policy` trust-chain taint tracking inside `ToolLoopOrchestrator`, `block_code_execution`, `mcp_security`, `mcp_end_user_permission`, and `mcp_jwt_signer` upstream token exchange.
- **Standalone Evaluation API & Micro-Dollar Telemetry (`migrations/0019`)**: Adds Data Plane `POST /v1/guardrails/apply`, expanded `/admin/v1/organizations/{orgID}/guardrails/simulate`, registry/binding/tool-policy/usage endpoints, declarative YAML seeding (`guardrails.definitions[]`, `guardrails.tool_policies[]`), scrubbed `guardrail.evaluate` OpenTelemetry spans, and `gateway_guardrail_evaluations_total`, `gateway_guardrail_errors_total`, and `gateway_guardrail_duration_ms` Prometheus metrics.
---
## Light Blue Dark & Light Operator Web UI Theme & Guardrails Console (`web/`, `internal/ui/`)
- **Dark & Light Theme System**: Adopts CSS-custom-property Obsidian Dark (`#000000` canvas, `#38bdf8` light-blue accent) and Pristine Slate Light (`#ffffff` canvas, `#0369a1` sky-blue accent) themes, backed by a CSP-compliant synchronous `ThemeContext` (`localStorage` persistence) and Header Dark/Light toggle.
- **4-Tab Guardrails Operator Console**: Adds lazy-loaded views for **Org Baseline Policy**, **Named Guardrails & Provider Registry** (including Starlark editor and tag matrix), **Tool Policies & Trust Matrix**, and **Pipeline Simulator (`/v1/guardrails/apply`) & Micro-Dollar Usage Dashboard**.
---
## Helm Chart (`0.3.0`) & Database Migrations (`0016`–`0019`)
- **Migrations `0016`–`0019`**: `0016_guardrail_policy_toxicity`, `0017_guardrail_registry_and_bindings`, `0018_tool_policies`, and `0019_guardrail_usage_metrics`.
- **Release Packaging**: Official Helm chart `0.3.0` (`appVersion: "1.8.0"`) and multi-arch container image `ghcr.io/goroute-ai/ai-gateway-go:v1.8.0`. See [Helm Chart Deployment](https://goroute-ai.github.io/docs/deploy/helm-chart).
---
## v1.7.0
URL: https://goroute-ai.github.io/docs/releases/v1.7.0
Markdown: https://goroute-ai.github.io/docs/releases/v1.7.0.md
Section: Control Plane Documentation > What changed
# v1.7.0
**Release Date:** 2026-11-14
**Theme:** Security & SSRF Remediations, Pure-SDK Parity, Operator Web UI Wiring, Helm Chart & Automated Release Packaging
---
## Security & SSRF Defense-in-Depth
- **Canonical Provider `BaseURL` & Preflight SSRF Enforcement** (`internal/adapter/`, `internal/server/`): Added canonical `ValidateAndResolveBaseURL` across native adapters (`openai`, `anthropic`, `mistral`, `ollama`, `gemini`, `azure`), `openaicompat` profiles (`ProfileRegistry` + `resolveEndpointBase`), and admin credential creation/rotation/validation handlers. Restricted private-network preflight probes to `platform_admin`, and hardened Gemini `isVertexExpressKey` to parsed hostname equality.
- **Isolated Untrusted Media Fetch Policy** (`internal/adapter/media.go`): Added `DefaultPublicMediaPolicy` and `FetchRemoteDataURIWithPolicy`, isolating end-user `image_url` and `document.url` fetches on `ollama` and `vertex` behind strict public-HTTPS `mediaPolicy` / `mediaClient` instances (`AllowPrivateNetworks: false`, `AllowInsecureHTTP: false`).
- **Response, Header & Concurrency Hardening** (`internal/server/`, `internal/auth/`, `internal/logger/`): Emitted strict `Content-Security-Policy: default-src 'none'; frame-ancestors 'none'` on non-UI routes, validated `/v1/audio/speech` `Content-Type`, sanitized `X-Request-Id` and fallback `Content-Type`, scoped `sha256.Sum256` JWT signature checks strictly to `RS256`/`ES256`, warned on short `auth.default_virtual_key` (< 32 chars), streamed multipart image uploads via `io.Copy`, and fixed a response-marshal race in `handleCreateBatch`.
---
## Go SDK (`pkg/gateway`), Contracts & Streaming Telemetry
- **WebSocket Realtime & Pure-SDK Modality Wrappers** (`pkg/gateway/`): Added `ConnectRealtime` (`*RealtimeSession` with `SendEvent`, `ReadEvent`, and `Close`), `WithHTTPClient`, and zero-internal-import typed SDK methods (`CreateEmbedding`, `CreateRerank`, `CreateModeration`, `CreateResponse`, `StreamResponse`, `CreateImage`, `CreateSpeech`, `TranscribeAudio`, and `GetSpendSummary`). See the [Go SDK API Reference](https://goroute-ai.github.io/docs/sdk/go-sdk-reference).
- **`/v1` + `/admin/v1` Spec Parity & Streaming Headers** (`cmd/specgen/`, `internal/server/`): Updated `cmd/specgen` to verify both `/v1` and `/admin/v1` route parity (`63` Unified OpenAPI paths), emitted `X-Gateway-Provider` and `X-Gateway-Model` before the first SSE chunk on streaming chat completions, and exposed `X-Gateway-*` telemetry headers via CORS `Access-Control-Expose-Headers`.
- **Multi-Replica HA & Tri-State Serialization** (`internal/auth/`, `internal/storage/`, `internal/server/`): Preserved 3-state allowlists (`null` = Inherit vs `[]` = Deny All) across Redis L2 `IdentityContext`, `Team`/`VirtualKey` JSON responses, and `optionalStringSlice` patch decoding (`PATCH`/`PUT` handlers), and ensured `PostgresRepository` mutations and `TEXT[] NOT NULL` bindings stay coherent across multi-replica pods.
---
## Routing & Operator Web UI (`web/`, `internal/ui/`)
- **6th Load-Balancing Strategy (`round-robin`)** (`internal/domain/`, `internal/router/`): Added `domain.StrategyRoundRobin` (`"round-robin"`) with per-`ModelGroupID` atomic rotation in `LoadBalancer.selectRoundRobin`. See [Routing & Load Balancing](https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing).
- **Control-Plane Contract Alignment**: Synced TypeScript models and views with `spent_micros`, 7 user/key roles (`platform_admin` through `service_account` / `viewer`), explicit 3-state `Inherit | Deny All | Custom List` allowlist controls, `round-robin` routing strategy, Pricing History modal, full Guardrails controls (`mode`, `scan_all_roles`, toxicity filter, content categories), `MCPTool.input_schema`, OAuth2/`static_headers` MCP config, `none | static | vault_jwt` A2A auth, inline `valid: false` credential validation errors, URL-encoded admin queries, and `<ScopedRoleNotice>` `403` handling across all 16 `sanitizeRuntimeConfig` sections.
- **SSE Playground & Nested SPA Assets**: Fixed SSE `event: error` detection and EOF decoder flushing, dynamically populated Playground model aliases from unlocked admin endpoints, removed empty assistant error placeholders, and switched favicon/logo assets to `/ui/*` paths within the `<= 420 KB` / `<= 125 KB` gzip bundle budget.
---
## Database Migrations (`0013`–`0015`) & Release Packaging
- **Migration `0013_user_limits_and_spend_indexes`**: Adds per-user `rpm_limit`, `tpm_limit`, `max_concurrent_requests`, `max_budget_micros`, `spent_micros`, `budget_reset_period`, and `budget_reset_at` columns to `users`, plus `(org_id, team_id)` and `(org_id, user_id)` indexes on `spend_logs`.
- **Migration `0014_budget_reset_outbox_user_scope`**: Expands `budget_reset_outbox_scope_check` to `CHECK (scope IN ('org', 'team', 'user', 'key'))`.
- **Migration `0015_team_org_concurrency`**: Adds `max_concurrent_requests` to `organizations` and `teams`.
- **Helm Chart & Automated Release Packaging** (`deployments/helm/goroute/`, `.github/workflows/release.yml`, `Makefile`): Added `make release-assets` for cross-platform archives (`darwin`/`linux`/`windows`, `amd64`/`arm64`), `checksums.txt`, Helm `.tgz` (`goroute` chart `0.2.0` / `appVersion: 1.7.0`), and Kubernetes bundle archives alongside multi-arch `ghcr.io/goroute-ai/ai-gateway-go` publishing with full OCI image labels. See [Helm Chart Deployment](https://goroute-ai.github.io/docs/deploy/helm-chart).
---
## v1.6.0
URL: https://goroute-ai.github.io/docs/releases/v1.6.0
Markdown: https://goroute-ai.github.io/docs/releases/v1.6.0.md
Section: Control Plane Documentation > What changed
# v1.6.0
**Release Date:** 2026-11-13
**Theme:** Maximum Provider & Model Compatibility, Modality FinOps, Production Test Suite & K8s Hardening
---
## Model Catalog, Provider Compatibility & Operation Enforcement
- **Upstream Catalog Parity & Capability Generalizations** (`cmd/catalog-sync/`, `internal/modelcatalog/`): Expanded `cmd/catalog-sync` and `internal/modelcatalog` with operation-level model support, modality pricing provenance, and explicit inferred capability generalizations (`internal/modelcatalog/generalizations.go`) without treating unverified catalog rates as verified for budgeted traffic.
- **Operation-Level Routing Enforcement & Certified OpenAI-Compatible Profiles** (`internal/router/`, `internal/adapter/openaicompat/`): Added `SupportedOperations` enforcement across router dispatch (`internal/router/reliability.go`) and expanded certified provider profiles (`internal/adapter/openaicompat/extended_profiles.go`, `validator.go`).
- **Multimodal Native Adapter & Responses API Extensions** (`internal/adapter/`): Expanded multimodal content, tool-only streaming, and audio/image/rerank wire handling across native adapters (`anthropic`, `bedrock`, `cohere`, `gemini`, `mistral`, `ollama`, `openai`, `vertex`) and `/v1/responses`.
---
## Storage, FinOps & Security Fixes
- **Migration `0012` & Modality Persistence** (`migrations/`): Added `migrations/0012_add_cache_write_and_operations.{up,down}.sql` for cache-write pricing and per-model operation declarations across PostgreSQL and in-memory repositories.
- **PostgreSQL `file_uploads` Query Fix** (`internal/storage/postgres.go`): Removed invalid `::uuid` casts on `VARCHAR(64)` `org_id`, `team_id`, and `user_id` columns in `CreateFile`, `GetFile`, `ListFiles`, and `DeleteFile`.
- **In-Memory Spend Outbox & Redis Breaker Fixes** (`internal/storage/`, `internal/cache/`): Deleted delivered entries from `MemoryRepository.spendOutbox` in `MarkSpendEventDelivered` to prevent $O(N^2)$ lock scans under sustained soak load, and treated `redis.Nil` cache misses as non-failures in `RedisFabric` circuit breakers.
---
## Production Testing, Docker & Kubernetes Hardening
- **Hardened Kubernetes Kustomize Manifests** (`deployments/k8s/`): Added `deployments/k8s/` with a 2-replica `gateway` deployment, non-root `securityContext` (`runAsNonRoot: true`, `runAsUser: 65532`, `readOnlyRootFilesystem: true`, `capabilities.drop: ["ALL"]`), `/readyz` + `/healthz` probes, and `postgres` + `redis` services. See [Kubernetes & Kustomize Deployment](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize).
- **Containerized Production Suite & Load/Canary Harnesses** (`deployments/`, `tests/`): Added `deployments/docker-compose.test.yml`, `tests/load/` (5-minute Go soak + `k6_gateway_smoke.js`), `tests/canary/` (`//go:build canary`), `tests/e2e/budget_concurrency_test.go`, `tests/e2e/modality_compat_test.go`, `tests/e2e/security_boundaries_test.go`, and `make test-prod-containers` (`gitleaks`, `trivy`, `govulncheck`, `k6`, `pgbench`, `redis-benchmark`).
---
## v1.5.0
URL: https://goroute-ai.github.io/docs/releases/v1.5.0
Markdown: https://goroute-ai.github.io/docs/releases/v1.5.0.md
Section: Control Plane Documentation > What changed
# v1.5.0
**Release Date:** 2026-11-13
**Theme:** Enterprise Security Hardening, Redis Cluster Support, Durable FinOps, Streaming Guardrails & API Parity
---
## Security & Multi-Tenant Authorization
- **Operator RBAC & Inbound JWT/OIDC** (`internal/auth/`, `internal/server/handlers_admin.go`): Added role-based operator authorization (`platform_admin`, `org_admin`, `org_viewer`) with organization-scoped control-plane enforcement, inbound JWT/OIDC verification, and disabled master-key data-plane fallback.
- **Fail-Closed Team Lookup & Allowlist Persistence** (`internal/storage/postgres.go`): Enforced fail-closed team lookup error handling and persisted all team allowlists (`allowed_models`, `allowed_mcp_servers`, `allowed_a2a_agents`, `allowed_client_apps`) in PostgreSQL.
- **HTTP Security Headers & CORS Middleware** (`internal/server/security_headers.go`): Added `SecurityHeadersMiddleware` (HSTS opt-in via `server.enable_hsts`, `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`, `Content-Security-Policy`) and safe-by-default `CORSMiddleware` (`server.cors_allowed_origins`, `server.cors_allow_credentials`).
- **Per-Organization File Storage Quota & Upload Concurrency Limits** (`internal/server/handlers_files.go`): Added `server.max_file_storage_bytes_per_org` (default `5 GiB`, `GATEWAY_MAX_FILE_STORAGE_BYTES_PER_ORG`) and `server.max_concurrent_file_uploads` (default `8`, `GATEWAY_MAX_CONCURRENT_FILE_UPLOADS`) with atomic in-flight reservation tracking (`413 storage_quota_exceeded`), bounded upload semaphore (`429 upload_concurrency_limit`), and explicit `MultipartForm.RemoveAll()` temporary spill-file cleanup.
- **Production Environment Hardening** (`internal/config/config.go`): Updated `Config.Validate()` to treat both `ENV=production` and `GATEWAY_ENV=production` as production signals and reject default admin, encryption, or JWT secrets.
- **Credential & Key Hygiene**: Restricted query-string credentials strictly to WebSocket upgrades (`/v1/realtime`), enforced expiry and revocation on cached virtual keys, and updated `golang.org/x/net` to `v0.62.0`.
- **CI & Supply-Chain Security**: Pinned all GitHub Actions workflows (`.github/workflows/ci.yml`, `.github/workflows/docker-publish.yml`) to immutable 40-character commit SHAs, added `.github/workflows/codeql.yml` for Go and JavaScript/TypeScript analysis, and added `.gitleaks.toml`.
---
## FinOps & Durable Micro-Dollar Accounting
- **Durable Spend Event Outbox** (`migrations/0008_add_spend_event_outbox.up.sql`): Added PostgreSQL spend event outbox with retryable delivery and idempotent Redis spend reconciliation keyed by event ID (`gw:spend:applied:{org}`).
- **Upper-Bound Budget Reservations & Lease Renewal** (`internal/finops/`, `internal/ratelimit/`): Added pre-dispatch upper-bound budget reservation with automatic lease renewal for long-running/streaming requests and unified modality budget gating.
- **Budget Reset Period Advancement & PG/Redis Sync** (`migrations/0011_budget_reset_outbox.up.sql`): Fixed `PostgresRepository.ResetExpiredBudgetsWithIDs`, synchronized entity `spent_micros`, and cleared Redis spend counters on period rollover.
- **Unpriced Usage & Modality Estimates**: Surfaced `unpriced_requests` and `unpriced_tokens` in spend summaries instead of silently recording zero cost, and improved preflight token estimates for modality requests.
---
## Routing, Reliability & Redis Cluster
- **Redis Cluster & Sentinel Support** (`internal/storage/`, `internal/cache/`): Upgraded `RedisFabric` to `go-redis` `UniversalClient` (`storage.redis_addrs`, `storage.redis_master_name`) with `{org:}` hash-tagged Lua keys across rate limiting and response caching.
- **Embeddings Reliability & Shared Circuit Breakers** (`internal/router/`): Wired `ExecuteEmbeddings` through `ReliabilityManager` retries/fallbacks, shared circuit-breaker failure counts and half-open probe leases in Redis across replicas, and fixed adaptive routing signals and least-busy reservation tracking.
- **Bounded Fallback Traversal & Streaming Retries**: Added cycle and depth limits to fallback group traversal and exponential backoff for streaming retries before first output token.
---
## Guardrails, MCP & A2A
- **All-Role Scanning, Streaming Redaction & `logging_only` Mode** (`internal/guardrails/stream_filter.go`, `migrations/0010_guardrail_mode.up.sql`): Expanded input scanning to all message roles, added output blocking and streaming chunk inspection/redaction, and added `logging_only` guardrail policy mode.
- **Fail-Closed Policy & Full Inference Coverage**: Enforced fail-closed guardrail policy lookups, added provider-neutral scanner hooks, and extended guardrail evaluation across all supported inference endpoints.
- **A2A & MCP Hardening** (`internal/a2a/`, `internal/mcp/`): Closed the A2A OpenAI bridge bypass, wired inbound/outbound A2A guardrails (`RouterService.SetGuardrails`), hardened MCP guardrail evaluation, and wired the native A2A gRPC server behind `server.grpc_port` (`GATEWAY_GRPC_PORT`).
---
## API Compatibility & Distributed Caching
- **OpenAI & Anthropic Wire Fidelity** (`internal/adapter/`): Added forwarding for `extra_body` and unknown top-level parameters (`ExtraParams`), Anthropic-formatted error envelopes on `/v1/messages`, streaming TTS (`/v1/audio/speech`), and `tools` forwarding on `/v1/responses`.
- **Canonical Response Cache Keys & Shared Invalidation** (`internal/cache/`): Keyed response cache entries by full canonical output-affecting request fields and propagated cache invalidation epochs across replicas via Redis.
---
## Observability & Embedded Operator Web UI
- **OTLP Distributed Tracing & Uniform Reservoir Sampling** (`internal/observability/`): Added optional OTLP/HTTP distributed tracing (`tracing.otlp_endpoint`, `gen_ai.*` span attributes) routed through the outbound SSRF policy, and replaced rolling-overwrite latency histograms with Vitter Algorithm R uniform reservoir sampling.
- **Structured Log Scrubbing & Protected Metrics** (`internal/logger/`): Added `logger.ScrubSecrets` to mask credentials in structured logs, protected `/metrics` with admin bearer authentication, and bounded Prometheus label cardinality.
- **Operator Web UI Hardening** (`web/`, `internal/ui/`): Repaired JSON field bindings, cleared React Query cache on Control Plane lock, self-hosted fonts (zero external font CDN calls), decoupled Playground/Agentic secrets, added `GET /admin/v1/mcp/tools`, and enforced CI `internal/ui/dist` parity.
---
## Go SDK Overview
URL: https://goroute-ai.github.io/docs/sdk/overview
Markdown: https://goroute-ai.github.io/docs/sdk/overview.md
Section: Go SDK Handbook > Build a Go application
# Go SDK Overview
Package: `github.com/goroute-ai/ai-gateway-go/pkg/gateway`. The SDK provides a Go client for chat requests and common gateway operations. It can start a gateway instance in the current process or connect to an HTTP gateway service.
## Choose a mode
| Mode | Constructor | What it does |
| --- | --- | --- |
| Embedded | `gateway.New(...)` | Loads gateway configuration, starts an in-process gateway server, and returns a client that talks to it over HTTP. |
| Remote | `gateway.NewClient(url, key)` or `gateway.WithRemote(url, key)` | Sends requests to an already running gateway service. |
The embedded mode does not require a separately deployed gateway service. It still runs the gateway server and uses HTTP internally; it is not a direct function call into the inference handler.
## Requirements and installation
The upstream module currently declares Go `1.26.0` in `go.mod`. Use a compatible Go toolchain:
```bash
go get github.com/goroute-ai/ai-gateway-go/pkg/gateway@v1.8.0
```
## Minimal request
```go
ctx := context.Background()
gw, err := gateway.New(
gateway.WithModelList(gateway.ModelEntry{
ModelName: "fast",
Model: "gemini/gemini-2.5-flash",
APIKey: "os.environ/GEMINI_API_KEY",
}),
gateway.WithDemoMocksIfEnvMissing(),
)
if err != nil {
return err
}
defer gw.Close()
result, err := gw.Chat(ctx, "fast", "Explain what an AI gateway does.")
if err != nil {
return err
}
fmt.Println(result.Content)
```
This is a function-body example; it assumes `context`, `fmt`, and the gateway package are imported and is inside a function that can return an error. The mock option provides a local fallback when the referenced environment key is unset. To verify a real provider call, configure a valid key and omit the mock option.
## Continue
- [Client configuration](https://goroute-ai.github.io/docs/sdk/client-configuration)
- [Chat and streaming](https://goroute-ai.github.io/docs/sdk/chat-and-streaming)
- [Virtual keys and FinOps](https://goroute-ai.github.io/docs/sdk/virtual-keys-and-finops)
- [MCP and A2A](https://goroute-ai.github.io/docs/sdk/mcp-and-a2a)
- [Public SDK types and methods](https://goroute-ai.github.io/docs/sdk/go-sdk-reference)
---
## Client Initialization & Configuration
URL: https://goroute-ai.github.io/docs/sdk/client-configuration
Markdown: https://goroute-ai.github.io/docs/sdk/client-configuration.md
Section: Go SDK Handbook > Build a Go application
# Client Initialization & Configuration
The `pkg/gateway` package uses functional options to configure an embedded gateway or a remote gateway client.
---
## 1. Initializing the Client
### `gateway.New(opts ...Option) (*Client, error)`
Initializes a client. If `WithRemote(...)` is provided, it connects to that remote service. Otherwise, it loads configuration and starts an embedded gateway server in the current process:
```go
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{ModelName: "fast", Model: "gemini/gemini-2.5-flash"},
),
gateway.WithLogLevel("info"),
)
if err != nil {
log.Fatalf("failed to initialize: %v", err)
}
defer gw.Close()
```
### `gateway.NewClient(baseURL, apiKey string) *Client`
Creates an HTTP client pointing to an already-running GoRoute server or container:
```go
client := gateway.NewClient("http://localhost:8080", "sk-gw-developer-key")
```
---
## 2. Configuration Options
### `WithModelList(entries ...ModelEntry) Option`
Configures the embedded router with declarative model aliases, fallbacks, and routing strategies.
```go
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "smart-router",
Model: "gemini/gemini-2.5-flash",
APIKey: "os.environ/GEMINI_API_KEY",
RoutingStrategy: "least-busy",
Fallbacks: []string{"backup-claude"},
RPM: 1000,
TPM: 500000,
},
gateway.ModelEntry{
ModelName: "backup-claude",
Model: "anthropic/claude-sonnet-4-20250514",
APIKey: "os.environ/ANTHROPIC_API_KEY",
},
),
)
```
#### `ModelEntry` Specification
| Field | Type | Description |
| :--- | :--- | :--- |
| `ModelName` | `string` | Client-facing alias (e.g. `"fast"`, `"reasoning"`, `"auto"`). |
| `Model` | `string` | Upstream provider/model identifier (e.g. `"gemini/gemini-2.5-flash"`, `"anthropic/claude-sonnet-4-20250514"`, `"openai/gpt-4.1-mini"`). |
| `APIKey` | `string` | Upstream API key string or environment reference (e.g. `"os.environ/GEMINI_API_KEY"`). |
| `APIBase` | `string` | Optional custom endpoint URL (e.g. for Ollama or private vLLM clusters: `"http://localhost:11434"`). |
| `RoutingStrategy`| `string` | Routing policy: `"weighted-shuffle"`, `"least-busy"`, `"latency-based"`, `"usage-based-headroom"`, `"lowest-cost"`, or `"round-robin"`. |
| `Fallbacks` | `[]string`| Ordered list of model aliases to invoke if this model returns 429, 503, or times out. |
| `Weight` | `int` | Integer weight for `"weighted-shuffle"` distribution. |
| `RPM` | `int` | Requests Per Minute limit enforced across this model entry. |
| `TPM` | `int` | Tokens Per Minute limit enforced across this model entry. |
| `SimulateStatus` | `int` | Optional fault injection HTTP code (e.g. `503`) to test circuit breakers and fallback chains offline. |
---
### `WithConfigFile(path string) Option`
Loads model definitions, rate limits, and security policies from an external YAML file:
```go
gw, err := gateway.New(
gateway.WithConfigFile("./config/gateway.yaml"),
)
```
**Sample YAML (`gateway.yaml`):**
```yaml
model_list:
- model_name: "production-fast"
model_params:
model: "gemini/gemini-2.5-flash"
api_key: "os.environ/GEMINI_API_KEY"
rpm: 1000
fallbacks: ["production-backup"]
- model_name: "production-backup"
model_params:
model: "openai/gpt-4.1-mini"
api_key: "os.environ/OPENAI_API_KEY"
```
---
### `WithDemoMocksIfEnvMissing() Option`
Enables built-in local mock HTTP servers for any `ModelEntry` whose environment variable API key is not present. This allows unit tests, integration tests, and local demos to run immediately without requiring live provider credentials or incurring costs.
```go
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{ModelName: "test-model", Model: "gemini/gemini-2.5-flash"},
),
gateway.WithDemoMocksIfEnvMissing(), // Mock provider handles requests locally
)
```
---
### Logging Options
Control internal gateway logging using Go standard library's `log/slog`:
- **`WithLogger(l *slog.Logger) Option`**: Provide a custom `slog.Logger` instance.
- **`WithSilentLogs() Option`**: Suppress all stdout/stderr logs (ideal for CLI tools or clean terminal outputs).
- **`WithLogLevel(level string) Option`**: Set minimum log level (`"debug"`, `"info"`, `"warn"`, `"error"`).
```go
customLogger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelDebug}))
gw, err := gateway.New(
gateway.WithLogger(customLogger),
)
```
---
### `WithRemote(baseURL, apiKey string) Option`
Redirects `gateway.New` to connect to a remote gateway daemon instead of starting an embedded router:
```go
gw, err := gateway.New(
gateway.WithRemote("https://ai-gateway.company.com", "sk-gw-production"),
)
```
---
### `WithHTTPClient(hc *http.Client) Option`
Configures a custom `*http.Client` for remote gateway HTTP requests and `/v1/realtime` WebSocket dialing (timeouts, custom TLS, transport pooling):
```go
customHTTP := &http.Client{Timeout: 45 * time.Second}
gw, err := gateway.New(
gateway.WithRemote("https://ai-gateway.company.com", "sk-gw-production"),
gateway.WithHTTPClient(customHTTP),
)
```
---
### `WithAllowPrivateNetworks() Option`
Permits embedded gateway connections to localhost or private network addresses (useful for local Ollama/LM Studio servers or in-process test mocks without enabling `WithDemoMocksIfEnvMissing()`).
---
## 3. Client Lifecycle & Scoping
### Dynamic Key Switching with `WithKey`
In multi-tenant backend services, create a shallow copy of the client with a tenant's specific Virtual API Key. The base connection and connection pooling are shared:
```go
// Base client initialized once at service startup
masterClient, _ := gateway.New(...)
// HTTP handler for tenant A
func TenantAHandler(w http.ResponseWriter, r *http.Request) {
tenantKey := r.Header.Get("X-Tenant-Key")
tenantClient := masterClient.WithKey(tenantKey)
// Chat requests are now governed by tenantKey's budget, rate limits, and model allowances
res, err := tenantClient.Chat(r.Context(), "fast", "Process tenant query")
...
}
```
### Teardown with `Close`
For embedded instances, `Close()` cleanly shuts down background workers, flushes remaining telemetry events, and terminates local mock servers:
```go
defer gw.Close()
```
*(For remote clients created via `NewClient`, `Close()` is a safe no-op).*
### Endpoint Inspection with `BaseURL`
Inspect the loopback HTTP URL generated for the in-process server (e.g. `http://127.0.0.1:54321`):
```go
fmt.Printf("Embedded gateway listening internally on: %s\n", gw.BaseURL())
```
---
## Chat and Streaming
URL: https://goroute-ai.github.io/docs/sdk/chat-and-streaming
Markdown: https://goroute-ai.github.io/docs/sdk/chat-and-streaming.md
Section: Go SDK Handbook > Build a Go application
# Chat and Streaming
The SDK exposes `Chat` for a single user prompt, `ChatWithMessages` for structured history, and `StreamChat` for receiving text deltas through a callback. The examples below are function-body snippets and assume `ctx context.Context`, an initialized `gw`, and the standard imports.
## Single-prompt chat
```go
result, err := gw.Chat(ctx, "fast", "Draft a short status update.",
gateway.WithSystem("You are a concise engineering assistant."),
gateway.WithTemperature(0.4),
gateway.WithMaxTokens(256),
)
if err != nil {
return err
}
fmt.Println(result.Content)
```
Available request options are `WithSystem`, `WithTemperature`, `WithMaxTokens`, and `WithReasoningEffort`. The upstream provider determines which options it supports.
## Multi-turn chat
```go
result, err := gw.ChatWithMessages(ctx, gateway.ChatRequest{
Model: "fast",
Messages: []gateway.Message{
{Role: "system", Content: "You are a concise site reliability engineer."},
{Role: "user", Content: "What should I check first after an HTTP 504?"},
{Role: "assistant", Content: "Check the gateway, upstream health, and timeout logs."},
{Role: "user", Content: "How can I narrow it down?"},
},
})
if err != nil {
return err
}
fmt.Println(result.Content)
```
## Stream text deltas
`StreamChat` calls `onDelta` as text arrives from the upstream stream. The callback can return an error to stop processing.
```go
result, err := gw.StreamChat(ctx, "fast", "Write a short haiku about Go.", func(delta string) error {
_, err := fmt.Print(delta)
return err
})
if err != nil {
return err
}
fmt.Printf("\nCost: %d µ$\n", result.CostMicros)
```
The gateway translates supported provider streaming responses into the SDK callback. Delivery timing and available content depend on the selected provider; no fixed latency or buffering guarantee is implied.
## Set a deadline
Pass a context with a timeout to bound the request and cancel work when the caller no longer needs the response:
```go
requestCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
result, err := gw.Chat(requestCtx, "fast", "Summarize this report.")
if err != nil {
return err
}
fmt.Println(result.Content)
```
## Pure-SDK Modality & Responses Wrappers
In `v1.7.0`, `pkg/gateway` provides self-contained public request and response structs (zero `internal/` package imports) for embeddings, reranking, moderation, the OpenAI Responses API, images, audio, and tokenization:
```go
// Vector embeddings
emb, err := gw.CreateEmbedding(ctx, gateway.EmbeddingRequest{
Model: "text-embedding-3-small",
Input: []string{"GoRoute AI Gateway"},
})
// Document reranking
reranked, err := gw.CreateRerank(ctx, gateway.RerankRequest{
Model: "rerank-v3.5",
Query: "zero-allocation Go proxy",
Documents: []any{"GoRoute routes inference in pure Go."},
})
// OpenAI Responses API (sync or streaming)
resp, err := gw.CreateResponse(ctx, gateway.ResponseRequest{
Model: "fast",
Input: "Summarize this diff in two bullets.",
})
_, err = gw.StreamResponse(ctx, gateway.ResponseRequest{
Model: "fast",
Input: "Stream a haiku about Go.",
}, func(delta string) error {
_, err := fmt.Print(delta)
return err
})
```
Additional pure-SDK modality methods include `CreateModeration`, `CreateImage`, `ImageEdit`, `ImageVariation`, `CreateSpeech`, `TranscribeAudio`, `Tokenize`, `CountTokens`, and stateful file/batch helpers (`UploadFile`, `ListFiles`, `GetFile`, `GetFileContent`, `DeleteFile`, `CreateBatch`, `ListBatches`, `GetBatch`, `CancelBatch`).
## Realtime WebSocket Sessions
Use `ConnectRealtime` to open a full-duplex WebSocket session to `/v1/realtime` using `*gateway.RealtimeSession`:
```go
session, err := gw.ConnectRealtime(ctx, "gpt-4o-realtime-preview")
if err != nil {
return err
}
defer session.Close()
if err := session.SendEvent(ctx, gateway.RealtimeEvent{
Type: "response.create",
}); err != nil {
return err
}
ev, err := session.ReadEvent(ctx)
if err != nil {
return err
}
fmt.Println("Received event:", ev.Type)
```
## Inspect result fields
`ChatResult` provides top-level fields for content, provider, model, request ID, cost in micro-dollars, routing time in microseconds, cache status, and whether a fallback was attempted. `Usage` and `Raw` expose internal domain types; prefer the top-level fields when you need a stable public SDK surface.
For function/tool definitions and the HTTP request shape, see [Tool Calling](https://goroute-ai.github.io/docs/gateway/endpoints/tool-calling). The current public `ChatRequest` struct does not expose a `tools` field.
---
## Virtual Keys and FinOps
URL: https://goroute-ai.github.io/docs/sdk/virtual-keys-and-finops
Markdown: https://goroute-ai.github.io/docs/sdk/virtual-keys-and-finops.md
Section: Go SDK Handbook > Build a Go application
# Virtual Keys and FinOps
The SDK provides methods for creating and revoking virtual keys and reading the spend summary. These are administrative operations: use a client configured with the gateway admin credential and protect it as a privileged object.
## Create a virtual key
```go
key, err := adminClient.CreateVirtualKey(ctx, gateway.VirtualKeyRequest{
Name: "payments-service",
OrgID: "org_default",
Role: "service_account",
MaxBudgetMicros: 25_000_000, // 25 USD
RPMLimit: 120,
TPMLimit: 100_000,
AllowedModels: []string{"fast"},
})
if err != nil {
return err
}
// Store key.Key in an approved secret store; do not log or commit it.
fmt.Println("Created key ID:", key.ID)
```
`MaxBudgetMicros` uses micro-dollars (`1 USD = 1,000,000 µ$`). The returned `key.Key` contains the new secret and is only available in the creation response. Store it securely.
## Use a scoped key
For server-controlled application code, create a request client using the virtual key:
```go
tenantClient := gateway.NewClient(gatewayURL, key.Key)
result, err := tenantClient.Chat(ctx, "fast", "Summarize the supplied text.")
```
`WithKey` can also copy a client with a different inference key. Keep any client carrying an admin credential inside trusted server code: the copy retains the original admin key for administrative methods.
## Revoke a key
```go
if err := adminClient.RevokeVirtualKey(ctx, key.ID); err != nil {
return err
}
```
The SDK calls the gateway's revoke endpoint. Subsequent authentication behavior depends on the gateway version and cache propagation; check the response and logs if a key still appears active.
## Read the spend summary
Use `GetSpendSummary` (added in `v1.7.0`) for a pure-SDK `*gateway.SpendSummaryResponse` with zero `internal/` package dependencies:
```go
summary, err := adminClient.GetSpendSummary(ctx)
if err != nil {
return err
}
fmt.Printf("Requests: %d; tokens: %d; cost: %d µ$ ($%.4f; unpriced: %d)\n",
summary.TotalRequests, summary.TotalTokens, summary.TotalCostMicros,
summary.TotalCostUSD, summary.UnpricedRequests)
fmt.Println("Spend by provider:", summary.SpendByProvider)
```
The SDK also exposes `UpdateVirtualKey`, `RotateVirtualKey`, `UpdateTeam`, `DeleteTeam`, `UpdateUser`, `DeleteUser`, and `GetConfig` using public SDK types, plus the legacy `SpendSummary` method (which returns an internal storage type). The stable JSON API is available at `GET /admin/v1/spend/summary`; see the [cost accounting guide](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger) and [Admin API reference](https://goroute-ai.github.io/docs/api/admin-api).
---
## MCP and A2A
URL: https://goroute-ai.github.io/docs/sdk/mcp-and-a2a
Markdown: https://goroute-ai.github.io/docs/sdk/mcp-and-a2a.md
Section: Go SDK Handbook > Build a Go application
# MCP and A2A
The SDK includes public configuration types for registering MCP servers and A2A agents. The gateway exposes the protocol routes for tool calls, agent messages, and tasks. Administrative registration methods require a client authenticated with an admin key and the corresponding server permissions.
## Register and sync an MCP server
```go
serverID, err := gw.AddMCPServer(ctx, gateway.MCPServerConfig{
Name: "Example tools",
Slug: "example-tools",
Transport: "streamable_http",
URL: "https://tools.example.com/mcp",
Enabled: true,
})
if err != nil {
return err
}
toolCount, err := gw.SyncMCPTools(ctx, serverID)
if err != nil {
return err
}
fmt.Printf("Synced %d tools\n", toolCount)
```
The gateway applies outbound network policy to the MCP server URL. Confirm the transport and endpoint path expected by your MCP server before registering it.
## List tools
`ListMCPTools` accepts an empty route for the virtual hub, a server slug, or a `toolsets/{slug}` route:
```go
tools, err := gw.ListMCPTools(ctx, "example-tools")
if err != nil {
return err
}
for _, tool := range tools {
fmt.Println(tool["name"], tool["description"])
}
```
The gateway also provides a plain REST bridge for clients that do not use JSON-RPC. See the [MCP overview](https://goroute-ai.github.io/docs/agents/mcp-overview), [hub and REST bridge](https://goroute-ai.github.io/docs/agents/mcp-hub-and-rest), and [MCP API reference](https://goroute-ai.github.io/docs/api/mcp-api).
## Register an A2A agent
```go
agentID, err := gw.AddA2AAgent(ctx, gateway.A2AAgentConfig{
Name: "Research agent",
Slug: "research",
Description: "Internal research agent",
EndpointURL: "https://agent.example.com",
})
if err != nil {
return err
}
fmt.Println("Registered agent:", agentID)
```
Use `GetWellKnownAgentCard(ctx)` to fetch the gateway's signed discovery card (`/.well-known/agent-card.json`), and consult the [A2A API reference](https://goroute-ai.github.io/docs/api/a2a-api) for message and task request schemas. The SDK's `SendA2AMessage` method currently accepts a protobuf type from an `internal` package, so applications outside the gateway module should use the HTTP API or another supported public client for dispatch.
## Security and permissions
- Use an admin client only for registration and management calls; use scoped virtual keys for application traffic.
- Allow only destinations that should receive gateway requests, and review the outbound SSRF/network policy.
- Disable Stdio MCP servers unless the gateway configuration and deployment are prepared to allow them.
---
## Go SDK API Reference
URL: https://goroute-ai.github.io/docs/sdk/go-sdk-reference
Markdown: https://goroute-ai.github.io/docs/sdk/go-sdk-reference.md
Section: Go SDK Handbook > Build a Go application
# Go SDK API Reference
Package: `github.com/goroute-ai/ai-gateway-go/pkg/gateway`
The signatures below reflect the public SDK types and methods. A few lower-level helpers expose types from Go's `internal` packages; those are omitted because applications outside the gateway module cannot import those types.
## Constructors and client lifecycle
```go
func New(opts ...Option) (*Client, error)
func NewClient(baseURL, apiKey string) *Client
func (c *Client) WithKey(virtualKey string) *Client
func (c *Client) Close()
func (c *Client) BaseURL() string
```
`New` starts an embedded instance unless configured with `WithRemote`. `NewClient` creates a client for an existing gateway. `Close` shuts down resources created for embedded mode; it is a no-op for a remote client.
## Client options
```go
func WithAllowPrivateNetworks() Option
func WithConfigFile(path string) Option
func WithDemoMocksIfEnvMissing() Option
func WithHTTPClient(hc *http.Client) Option
func WithLogLevel(level string) Option
func WithLogger(l *slog.Logger) Option
func WithModelList(entries ...ModelEntry) Option
func WithRemote(baseURL, apiKey string) Option
func WithSilentLogs() Option
func WithGuardrailScanner(name string, scanner GuardrailScanner, opts ...GuardrailOption) Option
```
### Guardrail Scanner Options & Types (`v1.8.0`)
```go
type GuardrailDecision string // "allow" | "block" | "flag" | "modify"
type GuardrailHook string // "pre_call" | "during_call" | "post_call" | "logging_only" | "pre_mcp_call" | "during_mcp_call" | "post_mcp_call"
type GuardrailScanner interface {
Scan(ctx context.Context, req GuardrailScanRequest) (GuardrailScanResult, error)
}
type GuardrailScannerFunc func(ctx context.Context, req GuardrailScanRequest) (GuardrailScanResult, error)
func WithGuardrailTimeout(d time.Duration) GuardrailOption
func WithGuardrailFailOpen(failOpen bool) GuardrailOption
func WithGuardrailHooks(hooks ...GuardrailHook) GuardrailOption
func WithGuardrailDefaultOn(defaultOn bool) GuardrailOption
func WithGuardrailPrecedence(precedence int) GuardrailOption
```
`Option` is an opaque option type accepted by `New`; its function signature uses an internal configuration type. Applications should pass the built-in options instead of constructing options directly.
## Chat methods
```go
func (c *Client) Chat(ctx context.Context, model, userPrompt string, opts ...RequestOption) (*ChatResult, error)
func (c *Client) ChatWithMessages(ctx context.Context, req ChatRequest, opts ...RequestOption) (*ChatResult, error)
func (c *Client) StreamChat(ctx context.Context, model, userPrompt string, onDelta func(delta string) error, opts ...RequestOption) (*ChatResult, error)
func (c *Client) ListModels(ctx context.Context) ([]string, error)
```
Request options:
```go
func WithSystem(systemPrompt string) RequestOption
func WithTemperature(temp float64) RequestOption
func WithMaxTokens(maxTokens int) RequestOption
func WithReasoningEffort(effort string) RequestOption
```
Pass these helpers to `Chat` or `StreamChat`. The SDK exposes the option type for use with its built-in helpers; applications should not implement custom options because the function signature depends on an internal request type.
## Core data types
```go
type ModelEntry struct {
ModelName string
Model string
APIKey string
APIBase string
RoutingStrategy string
Fallbacks []string
Weight int
RPM int
TPM int
SimulateStatus int
}
type ChatResult struct {
Content string
ReasoningContent string
Provider string
Model string
RequestID string
CostMicros int64
RoutingMicros int64
CacheStatus string
RouteReason string
FallbackAttempted bool
// Usage and Raw are omitted because their concrete types come from internal packages.
}
type ChatRequest struct {
Model string
Messages []Message
Temperature *float64
TopP *float64
MaxTokens *int
Stop []string
Stream bool
ReasoningEffort string
User string
}
type Message struct {
Role string
Content string
Name string
ToolCallID string
ToolCalls []ToolCall
}
type ToolCall struct {
ID string
Type string
Function FunctionCall
}
type FunctionCall struct {
Name string
Arguments string
}
```
The `ChatResult` excerpt shows the stable top-level fields most applications need. It also has `Usage` and `Raw` fields whose concrete types come from internal packages; prefer top-level fields when you need to avoid depending on those implementation types. JSON tags are omitted here for readability; consult the source for serialization details.
## Modality, Responses, Files, Batches & Realtime (`v1.7.0`)
Pure-SDK methods using self-contained public structs (no `internal/` imports):
```go
// Embeddings, Rerank, Moderation, Responses
func (c *Client) CreateEmbedding(ctx context.Context, req EmbeddingRequest) (*EmbeddingResponse, error)
func (c *Client) CreateRerank(ctx context.Context, req RerankRequest) (*RerankResponse, error)
func (c *Client) CreateModeration(ctx context.Context, req ModerationRequest) (*ModerationResponse, error)
func (c *Client) CreateResponse(ctx context.Context, req ResponseRequest) (*ResponseOutput, error)
func (c *Client) StreamResponse(ctx context.Context, req ResponseRequest, onDelta func(delta string) error) (*ResponseOutput, error)
// Images & Audio
func (c *Client) CreateImage(ctx context.Context, req ImageGenerationRequest) (*ImageResponse, error)
func (c *Client) ImageEdit(ctx context.Context, req ImageEditRequest) (*ImageResponse, error)
func (c *Client) ImageVariation(ctx context.Context, req ImageVariationRequest) (*ImageResponse, error)
func (c *Client) CreateSpeech(ctx context.Context, req SpeechRequest) ([]byte, error)
func (c *Client) TranscribeAudio(ctx context.Context, req AudioTranscriptionRequest) (*AudioTranscriptionResult, error)
// Tokenization & Anthropic Messages pass-through
func (c *Client) Tokenize(ctx context.Context, req TokenizeRequest) (*TokenizeResponse, error)
func (c *Client) CountTokens(ctx context.Context, req CountTokensRequest) (*CountTokensResponse, error)
func (c *Client) Messages(ctx context.Context, req map[string]any) (map[string]any, error)
func (c *Client) StreamMessages(ctx context.Context, req map[string]any, onDelta func(delta string) error) (map[string]any, error)
// Stateful Files & Batches
func (c *Client) UploadFile(ctx context.Context, req FileUploadRequest) (*FileObject, error)
func (c *Client) ListFiles(ctx context.Context, purpose string) ([]FileObject, error)
func (c *Client) GetFile(ctx context.Context, fileID string) (*FileObject, error)
func (c *Client) GetFileContent(ctx context.Context, fileID string) ([]byte, error)
func (c *Client) DeleteFile(ctx context.Context, fileID string) error
func (c *Client) CreateBatch(ctx context.Context, req BatchCreateRequest) (*BatchObject, error)
func (c *Client) ListBatches(ctx context.Context) ([]BatchObject, error)
func (c *Client) GetBatch(ctx context.Context, batchID string) (*BatchObject, error)
func (c *Client) CancelBatch(ctx context.Context, batchID string) (*BatchObject, error)
// WebSocket Realtime Sessions
func (c *Client) ConnectRealtime(ctx context.Context, model string) (*RealtimeSession, error)
func (s *RealtimeSession) SendEvent(ctx context.Context, ev RealtimeEvent) error
func (s *RealtimeSession) ReadEvent(ctx context.Context) (*RealtimeEvent, error)
func (s *RealtimeSession) Close() error
```
Public modality/realtime structs in `pkg/gateway`: `EmbeddingRequest`, `EmbeddingData`, `EmbeddingResponse`, `UsageInfo`, `RerankRequest`, `RerankResult`, `RerankUsage`, `RerankResponse`, `ModerationRequest`, `ModerationResult`, `ModerationResponse`, `ResponseRequest`, `ResponseContentPart`, `ResponseOutputItem`, `ResponseOutput`, `ImageGenerationRequest`, `ImageEditRequest`, `ImageVariationRequest`, `ImageData`, `ImageResponse`, `SpeechRequest`, `AudioTranscriptionRequest`, `AudioTranscriptionResult`, `TokenizeRequest`, `TokenizeResponse`, `CountTokensMessage`, `CountTokensRequest`, `CountTokensResponse`, `FileUploadRequest`, `FileObject`, `BatchCreateRequest`, `BatchCounts`, `BatchObject`, `RealtimeSessionConfig`, `RealtimeEvent`, and `RealtimeSession`.
## Virtual keys, governance, FinOps & protocol registration
```go
func (c *Client) CreateVirtualKey(ctx context.Context, req VirtualKeyRequest) (*VirtualKeyInfo, error)
func (c *Client) UpdateVirtualKey(ctx context.Context, keyID string, req VirtualKeyUpdateRequest) (*VirtualKeyConfig, error)
func (c *Client) RotateVirtualKey(ctx context.Context, keyID string) (*VirtualKeyRotateResponse, error)
func (c *Client) RevokeVirtualKey(ctx context.Context, keyID string) error
func (c *Client) UpdateTeam(ctx context.Context, teamID string, req TeamUpdateRequest) (*TeamConfig, error)
func (c *Client) DeleteTeam(ctx context.Context, teamID string) error
func (c *Client) UpdateUser(ctx context.Context, userID string, req UserUpdateRequest) (*UserConfig, error)
func (c *Client) DeleteUser(ctx context.Context, userID string) error
func (c *Client) GetSpendSummary(ctx context.Context) (*SpendSummaryResponse, error)
func (c *Client) GetConfig(ctx context.Context) (map[string]any, error)
func (c *Client) AddMCPServer(ctx context.Context, srv MCPServerConfig) (string, error)
func (c *Client) SyncMCPTools(ctx context.Context, serverID string) (int, error)
func (c *Client) ListMCPTools(ctx context.Context, route string) ([]map[string]any, error)
func (c *Client) AddA2AAgent(ctx context.Context, agent A2AAgentConfig) (string, error)
func (c *Client) GetWellKnownAgentCard(ctx context.Context) (map[string]any, error)
```
The corresponding public configuration and governance types are `VirtualKeyRequest`, `VirtualKeyInfo`, `VirtualKeyConfig`, `VirtualKeyUpdateRequest`, `VirtualKeyRotateResponse`, `TeamConfig`, `TeamUpdateRequest`, `UserConfig`, `UserUpdateRequest`, `SpendSummaryResponse`, `MCPServerConfig`, and `A2AAgentConfig`. Use the SDK feature guides for examples and required server permissions:
- [Client configuration](https://goroute-ai.github.io/docs/sdk/client-configuration)
- [Chat and streaming](https://goroute-ai.github.io/docs/sdk/chat-and-streaming)
- [Virtual keys and FinOps](https://goroute-ai.github.io/docs/sdk/virtual-keys-and-finops)
- [MCP and A2A](https://goroute-ai.github.io/docs/sdk/mcp-and-a2a)
---
## API Reference Index
URL: https://goroute-ai.github.io/docs/api/overview
Markdown: https://goroute-ai.github.io/docs/api/overview.md
Section: API Reference > API reference
# API Reference Index
Use this page when you already know which kind of connection you need. The API reference is grouped by request handling, administration, MCP tools, and A2A agents. If you are new to GoRoute, begin with the [overview](https://goroute-ai.github.io/docs/start/overview) before choosing an API.
GoRoute exposes these capabilities through HTTP, WebSocket, and gRPC interfaces. The catalog below lets you search routes and narrow them by API surface.
> [!NOTE]
> All data plane, MCP, and A2A routes require a tenant-scoped Virtual API Key passed via `Authorization: Bearer sk-gw-...`.
> All administrative control plane routes require the Master Admin Key passed via `Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY`.
---
## Interactive Endpoint Catalog
Search and filter every HTTP and WebSocket route exposed by GoRoute AI Gateway across `/v1/*`, `/openai/v1/*`, `/anthropic/v1/*`, `/mcp*`, `/a2a/v1/*`, and `/admin/v1/*`. The interactive catalog documents **112 endpoints across 7 API surfaces**, while the unified OpenAPI specification (`gateway.openapi.yaml`) covers **73 paths** (`24` `/v1` + `49` `/admin/v1`) with automated parity enforced by `cmd/specgen`.
### Gateway Endpoints (112 Documented)
| Method | Path | Surface | Auth | Summary |
| :--- | :--- | :--- | :--- | :--- |
| `POST` | `/v1/chat/completions` | Data Plane (/v1) | Virtual API Key | Unified OpenAI-compatible chat completions with streaming SSE, tool calling, reasoning, and FinOps headers. |
| `POST` | `/v1/completions` | Data Plane (/v1) | Virtual API Key | Legacy text completions endpoint with streaming support. |
| `POST` | `/v1/embeddings` | Data Plane (/v1) | Virtual API Key | Generate vector embeddings across OpenAI, Cohere, Gemini, Bedrock, Vertex, and dialect providers. |
| `GET` | `/v1/models` | Data Plane (/v1) | Virtual API Key | List models accessible to the authenticated virtual key or tenant. |
| `POST` | `/v1/rerank` | Data Plane (/v1) | Virtual API Key | Rerank candidate documents against a query (Cohere, Voyage, Jina, Together, Fireworks, Novita, NVIDIA). |
| `POST` | `/v1/moderations` | Data Plane (/v1) | Virtual API Key | Classify text inputs for policy and content safety violations. |
| `POST` | `/v1/messages` | Data Plane (/v1) | Virtual API Key | Anthropic Messages API ingress supporting Claude SDKs and extended thinking. |
| `POST` | `/v1/messages/count_tokens` | Data Plane (/v1) | Virtual API Key | Count input tokens using the Anthropic Messages token-counting schema. |
| `POST` | `/v1/tokenize` | Data Plane (/v1) | Virtual API Key | Pre-flight token count estimation for request payloads. |
| `POST` | `/v1/audio/transcriptions` | Data Plane (/v1) | Virtual API Key | Speech-to-text transcription via multipart audio upload. |
| `POST` | `/v1/audio/translations` | Data Plane (/v1) | Virtual API Key | Speech-to-English audio translation. |
| `POST` | `/v1/audio/speech` | Data Plane (/v1) | Virtual API Key | Text-to-speech audio synthesis returning binary audio streams. |
| `POST` | `/v1/images/generations` | Data Plane (/v1) | Virtual API Key | Generate images from text prompts. |
| `POST` | `/v1/images/edits` | Data Plane (/v1) | Virtual API Key | Edit or inpaint an existing image with a prompt and optional mask. |
| `POST` | `/v1/images/variations` | Data Plane (/v1) | Virtual API Key | Create visual variations of an uploaded source image. |
| `POST` | `/v1/files` | Data Plane (/v1) | Virtual API Key | Upload a file for batch processing or assistants. |
| `GET` | `/v1/files` | Data Plane (/v1) | Virtual API Key | List uploaded files scoped to the tenant. |
| `GET` | `/v1/files/{file_id}` | Data Plane (/v1) | Virtual API Key | Retrieve metadata for a specific uploaded file. |
| `GET` | `/v1/files/{file_id}/content` | Data Plane (/v1) | Virtual API Key | Download the raw contents of an uploaded file. |
| `DELETE` | `/v1/files/{file_id}` | Data Plane (/v1) | Virtual API Key | Delete an uploaded file. |
| `POST` | `/v1/batches` | Data Plane (/v1) | Virtual API Key | Create an asynchronous batch inference job. |
| `GET` | `/v1/batches` | Data Plane (/v1) | Virtual API Key | List asynchronous batch jobs for the tenant. |
| `GET` | `/v1/batches/{batch_id}` | Data Plane (/v1) | Virtual API Key | Inspect status and output file IDs for a batch job. |
| `POST` | `/v1/batches/{batch_id}/cancel` | Data Plane (/v1) | Virtual API Key | Cancel an in-progress batch job. |
| `POST` | `/v1/responses` | Data Plane (/v1) | Virtual API Key | OpenAI Responses API for stateful multi-turn reasoning and tool execution. |
| `WS` | `/v1/realtime` | Data Plane (/v1) | Virtual API Key | Full-duplex WebSocket proxy for low-latency realtime audio and text sessions. |
| `POST` | `/v1/guardrails/apply` | Data Plane (/v1) | Virtual API Key | Evaluate input/output text or tool calls directly against Pipeline v2 guardrails. |
| `POST` | `/openai/v1/responses` | OpenAI Direct (/openai/v1) | Virtual API Key | Direct OpenAI-prefixed Responses API alias. |
| `WS` | `/openai/v1/realtime` | OpenAI Direct (/openai/v1) | Virtual API Key | Direct OpenAI-prefixed Realtime WebSocket alias. |
| `POST` | `/anthropic/v1/messages` | Anthropic Native (/anthropic/v1) | Virtual API Key | Native Anthropic Messages endpoint for Claude Code and Anthropic SDKs. |
| `POST` | `/anthropic/v1/messages/count_tokens` | Anthropic Native (/anthropic/v1) | Virtual API Key | Native Anthropic token-counting endpoint. |
| `GET` | `/mcp/subscriptions/listen` | MCP (/mcp*) | Virtual API Key | SSE stream for MCP resource and tool list change notifications. |
| `POST` | `/mcp` | MCP (/mcp*) | Virtual API Key | MCP Virtual Hub JSON-RPC endpoint aggregating tools across upstream servers. |
| `POST` | `/mcp/{serverSlug}` | MCP (/mcp*) | Virtual API Key | Scoped MCP JSON-RPC endpoint targeting a single registered MCP server. |
| `POST` | `/mcp/toolsets/{toolsetSlug}` | MCP (/mcp*) | Virtual API Key | Curated MCP toolset endpoint exposing an approved subset of tools. |
| `GET` | `/mcp-rest/tools/list` | MCP (/mcp*) | Virtual API Key | Plain REST bridge listing available MCP tools without JSON-RPC framing. |
| `POST` | `/mcp-rest/tools/call` | MCP (/mcp*) | Virtual API Key | Plain REST bridge invoking an MCP tool by name and arguments. |
| `GET` | `/v1/goroute/mcp/registry` | MCP (/mcp*) | Virtual API Key | Progressive tool catalog registry feed returning compact tool handles. |
| `GET` | `/.well-known/agent-card.json` | A2A (/a2a/v1) | Public | Signed gateway Agent Card JSON for A2A capability discovery. |
| `POST` | `/a2a/v1/agents/{agentSlug}/` | A2A (/a2a/v1) | Virtual API Key | Send an A2A message/task to a registered agent. |
| `GET` | `/a2a/v1/agents/{agentSlug}/tasks` | A2A (/a2a/v1) | Virtual API Key | List A2A tasks for the specified agent. |
| `GET` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}` | A2A (/a2a/v1) | Virtual API Key | Retrieve current state and artifacts of an A2A task. |
| `POST` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}:cancel` | A2A (/a2a/v1) | Virtual API Key | Cancel an active A2A task. |
| `GET` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}:subscribe` | A2A (/a2a/v1) | Virtual API Key | Subscribe-first SSE stream for live A2A task status and artifact updates. |
| `POST` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}:subscribe` | A2A (/a2a/v1) | Virtual API Key | POST variant for subscribing to an A2A task SSE stream. |
| `POST` | `/admin/v1/organizations` | Admin Control Plane (/admin/v1) | Master Admin Key | Create a new tenant organization. |
| `GET` | `/admin/v1/organizations` | Admin Control Plane (/admin/v1) | Master Admin Key | List all organizations. |
| `POST` | `/admin/v1/organizations/{orgID}/subscriptions` | Admin Control Plane (/admin/v1) | Master Admin Key | Set provider subscription access for an organization. |
| `GET` | `/admin/v1/organizations/{orgID}/subscriptions` | Admin Control Plane (/admin/v1) | Master Admin Key | Get provider subscriptions for an organization. |
| `POST` | `/admin/v1/organizations/{orgID}/models` | Admin Control Plane (/admin/v1) | Master Admin Key | Configure model allowlist/denylist policies for an organization. |
| `GET` | `/admin/v1/organizations/{orgID}/models` | Admin Control Plane (/admin/v1) | Master Admin Key | Retrieve model policies for an organization. |
| `POST` | `/admin/v1/organizations/{orgID}/guardrails` | Admin Control Plane (/admin/v1) | Master Admin Key | Set input/output security guardrail policies for an organization. |
| `GET` | `/admin/v1/organizations/{orgID}/guardrails` | Admin Control Plane (/admin/v1) | Master Admin Key | Get configured guardrail policies for an organization. |
| `POST` | `/admin/v1/organizations/{orgID}/guardrails/simulate` | Admin Control Plane (/admin/v1) | Master Admin Key | Dry-run guardrail rules and Pipeline v2 drivers against a sample payload. |
| `POST` | `/admin/v1/organizations/{orgID}/guardrail-definitions` | Admin Control Plane (/admin/v1) | Master Admin Key | Create a named guardrail definition backed by one of 19 drivers, Starlark, or webhooks. |
| `GET` | `/admin/v1/organizations/{orgID}/guardrail-definitions` | Admin Control Plane (/admin/v1) | Master Admin Key | List named guardrail definitions configured for an organization. |
| `GET` | `/admin/v1/guardrail-definitions/{guardrailID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Retrieve a named guardrail definition by ID. |
| `PUT` | `/admin/v1/guardrail-definitions/{guardrailID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Replace configuration and credentials for a guardrail definition. |
| `PATCH` | `/admin/v1/guardrail-definitions/{guardrailID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Partially update a named guardrail definition. |
| `DELETE` | `/admin/v1/guardrail-definitions/{guardrailID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Delete a named guardrail definition. |
| `POST` | `/admin/v1/guardrail-definitions/{guardrailID}/status` | Admin Control Plane (/admin/v1) | Master Admin Key | Enable or disable a named guardrail definition. |
| `POST` | `/admin/v1/guardrail-bindings` | Admin Control Plane (/admin/v1) | Master Admin Key | Bind a guardrail definition to an Org, Team, VirtualKey, EndUser, ModelGroup, or Tag. |
| `GET` | `/admin/v1/organizations/{orgID}/guardrail-bindings` | Admin Control Plane (/admin/v1) | Master Admin Key | List hierarchical guardrail bindings for an organization. |
| `DELETE` | `/admin/v1/guardrail-bindings/{bindingID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Remove a guardrail binding. |
| `POST` | `/admin/v1/organizations/{orgID}/tool-policies` | Admin Control Plane (/admin/v1) | Master Admin Key | Create or update tool permission and cross-turn taint trust policies. |
| `GET` | `/admin/v1/organizations/{orgID}/tool-policies` | Admin Control Plane (/admin/v1) | Master Admin Key | List tool trust policies configured for an organization. |
| `DELETE` | `/admin/v1/organizations/{orgID}/tool-policies/{policyID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Delete a tool trust policy. |
| `GET` | `/admin/v1/organizations/{orgID}/guardrail-usage` | Admin Control Plane (/admin/v1) | Master Admin Key | Query micro-dollar guardrail usage rollups and evaluation counts. |
| `POST` | `/admin/v1/teams` | Admin Control Plane (/admin/v1) | Master Admin Key | Create a team within an organization. |
| `GET` | `/admin/v1/teams` | Admin Control Plane (/admin/v1) | Master Admin Key | List teams. |
| `POST` | `/admin/v1/users` | Admin Control Plane (/admin/v1) | Master Admin Key | Create a user. |
| `GET` | `/admin/v1/users` | Admin Control Plane (/admin/v1) | Master Admin Key | List users. |
| `POST` | `/admin/v1/keys` | Admin Control Plane (/admin/v1) | Master Admin Key | Issue a scoped Virtual API Key (sk-gw-...) with rate and budget limits. |
| `GET` | `/admin/v1/keys` | Admin Control Plane (/admin/v1) | Master Admin Key | List issued Virtual API Keys. |
| `POST` | `/admin/v1/keys/{keyID}/revoke` | Admin Control Plane (/admin/v1) | Master Admin Key | Revoke a Virtual API Key. |
| `DELETE` | `/admin/v1/keys/{keyID}` | Admin Control Plane (/admin/v1) | Master Admin Key | Delete/revoke a Virtual API Key. |
| `POST` | `/admin/v1/credentials` | Admin Control Plane (/admin/v1) | Master Admin Key | Store an upstream provider credential encrypted with AES-256-GCM. |
| `GET` | `/admin/v1/credentials` | Admin Control Plane (/admin/v1) | Master Admin Key | List stored provider credentials (secrets redacted). |
| `POST` | `/admin/v1/credentials/validate` | Admin Control Plane (/admin/v1) | Master Admin Key | Preflight-validate a provider credential before storing. |
| `POST` | `/admin/v1/credentials/{credID}/rotate` | Admin Control Plane (/admin/v1) | Master Admin Key | Rotate a stored provider credential. |
| `POST` | `/admin/v1/credentials/{credID}/validate` | Admin Control Plane (/admin/v1) | Master Admin Key | Live-validate an existing stored credential against the upstream provider. |
| `POST` | `/admin/v1/model-groups` | Admin Control Plane (/admin/v1) | Master Admin Key | Create a logical model alias group with routing strategy. |
| `GET` | `/admin/v1/model-groups` | Admin Control Plane (/admin/v1) | Master Admin Key | List model groups. |
| `POST` | `/admin/v1/deployments` | Admin Control Plane (/admin/v1) | Master Admin Key | Register an upstream model deployment. |
| `GET` | `/admin/v1/deployments` | Admin Control Plane (/admin/v1) | Master Admin Key | List configured deployments. |
| `GET` | `/admin/v1/catalogue` | Admin Control Plane (/admin/v1) | Master Admin Key | List discovered models and versioned pricing in the global catalogue. |
| `POST` | `/admin/v1/catalogue/sync` | Admin Control Plane (/admin/v1) | Master Admin Key | Trigger immediate upstream model discovery sync. |
| `POST` | `/admin/v1/catalogue/import` | Admin Control Plane (/admin/v1) | Master Admin Key | Import model catalogue and pricing definitions. |
| `POST` | `/admin/v1/catalogue/pricing` | Admin Control Plane (/admin/v1) | Master Admin Key | Update micro-dollar pricing for a model. |
| `GET` | `/admin/v1/catalogue/pricing` | Admin Control Plane (/admin/v1) | Master Admin Key | List versioned model pricing history. |
| `GET` | `/admin/v1/observability/metrics` | Admin Control Plane (/admin/v1) | Master Admin Key | Retrieve JSON snapshot of gateway telemetry and cache/routing metrics. |
| `GET` | `/admin/v1/health/deployments` | Admin Control Plane (/admin/v1) | Master Admin Key | Inspect circuit breaker and health status of all deployments. |
| `POST` | `/admin/v1/health/probe` | Admin Control Plane (/admin/v1) | Master Admin Key | Trigger active health probes across deployments. |
| `POST` | `/admin/v1/cache/invalidate` | Admin Control Plane (/admin/v1) | Master Admin Key | Invalidate L1/L2 exact-match response cache entries. |
| `GET` | `/admin/v1/spend/summary` | Admin Control Plane (/admin/v1) | Master Admin Key | Aggregated micro-dollar spend summary by org, team, or key. |
| `GET` | `/admin/v1/spend/logs` | Admin Control Plane (/admin/v1) | Master Admin Key | Paginated per-request FinOps spend ledger. |
| `GET` | `/admin/v1/audit/logs` | Admin Control Plane (/admin/v1) | Master Admin Key | Control-plane governance audit log entries. |
| `POST` | `/admin/v1/mcp/servers` | Admin Control Plane (/admin/v1) | Master Admin Key | Register an upstream MCP server (HTTP or sandboxed stdio). |
| `GET` | `/admin/v1/mcp/servers` | Admin Control Plane (/admin/v1) | Master Admin Key | List registered MCP servers. |
| `POST` | `/admin/v1/mcp/servers/{serverID}/sync` | Admin Control Plane (/admin/v1) | Master Admin Key | Sync tool definitions from an MCP server. |
| `POST` | `/admin/v1/mcp/toolsets` | Admin Control Plane (/admin/v1) | Master Admin Key | Create a curated MCP toolset. |
| `GET` | `/admin/v1/mcp/toolsets` | Admin Control Plane (/admin/v1) | Master Admin Key | List curated MCP toolsets. |
| `GET` | `/admin/v1/mcp/tools` | Admin Control Plane (/admin/v1) | Master Admin Key | List all synced MCP tools. |
| `POST` | `/admin/v1/mcp/credentials` | Admin Control Plane (/admin/v1) | Master Admin Key | Store BYOK credentials for an MCP server. |
| `POST` | `/admin/v1/a2a/agents` | Admin Control Plane (/admin/v1) | Master Admin Key | Register an A2A agent. |
| `GET` | `/admin/v1/a2a/agents` | Admin Control Plane (/admin/v1) | Master Admin Key | List registered A2A agents. |
| `GET` | `/admin/v1/a2a/agents/{id}` | Admin Control Plane (/admin/v1) | Master Admin Key | Get details for a registered A2A agent. |
| `POST` | `/admin/v1/a2a/agents/{id}/kill-switch` | Admin Control Plane (/admin/v1) | Master Admin Key | Toggle the emergency kill-switch for an A2A agent. |
| `POST` | `/admin/v1/a2a/push-configs` | Admin Control Plane (/admin/v1) | Master Admin Key | Configure webhook push notifications for A2A tasks. |
| `GET` | `/healthz` | System & Telemetry | Public | Liveness probe returning service OK status. |
| `GET` | `/readyz` | System & Telemetry | Public | Readiness probe verifying PostgreSQL and Redis connectivity. |
| `GET` | `/metrics` | System & Telemetry | Metrics Auth | Prometheus exposition endpoint for gateway, cache, and upstream HTTP metrics. |
---
## API Reference Guides
```mermaid
flowchart TD
Index["GoRoute API Surface"] --> DP["Data Plane API
(/v1/*)"]
Index --> Admin["Admin Control Plane
(/admin/v1/*)"]
Index --> MCP["MCP Virtual Hub
(/mcp, /mcp-rest)"]
Index --> A2A["A2A Protocol Service
(/a2a/v1/*, gRPC)"]
DP --> DP1["Chat, Completions and Reasoning"]
DP --> DP2["Audio, Images, Embeddings and Rerank"]
DP --> DP3["Stateful Files, Batches and Realtime"]
Admin --> AD1["Orgs, Teams, Users and Virtual Keys"]
Admin --> AD2["Credentials Vault, Deployments and Pricing"]
Admin --> AD3["FinOps Ledgers, Guardrails and Health Probes"]
MCP --> M1["tools/list Aggregator and tools/call"]
MCP --> M2["Plain REST Bridge and Progressive Handles"]
A2A --> A1["Signed Agent Cards and Message Dispatch"]
A2A --> A2["SSE Task Streams, Native gRPC and Bridge"]
```
| Specification Guide | Endpoints Base | Core Coverage |
| :--- | :--- | :--- |
| **[Gateway Data Plane API](https://goroute-ai.github.io/docs/api/gateway-api)** | `/v1/*` | All 24 primary `/v1/*` paths: Chat Completions, Text Completions, Embeddings, Models, Rerank, Moderations, Anthropic Messages, Tokenization, Standalone Guardrails (`/v1/guardrails/apply`), Audio (STT & TTS), Images, Files, Batches, Responses, and Realtime WebSockets. |
| **[Admin Control Plane API](https://goroute-ai.github.io/docs/api/admin-api)** | `/admin/v1/*` | All management groups: Organizations, Provider Subscriptions, Model Policies, Guardrails & Simulator, Teams, Users, Virtual Keys, Credentials Vault, Model Groups, Deployments, Catalogue/Pricing, Health/Cache, and FinOps Ledgers. |
| **[Model Context Protocol (MCP) API](https://goroute-ai.github.io/docs/api/mcp-api)** | `/mcp/*`, `/mcp-rest/*` | Virtual Hub aggregator, Scoped server execution, Curated toolsets, Plain REST tool bridge, and Progressive Tool Discovery catalog feed. |
| **[Agent-to-Agent (A2A) API](https://goroute-ai.github.io/docs/api/a2a-api)** | `/a2a/v1/*`, `gRPC` | Signed Agent Cards (`/.well-known/agent-card.json`), dual-binding message dispatch, Subscribe-First SSE task streaming, native gRPC service, and OpenAI SDK compatibility bridge. |
---
## Authentication Schemes
```http
# Data Plane, MCP, and A2A requests (Virtual API Key)
Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e
# Admin Control Plane requests (Master Admin Key)
Authorization: Bearer sk-gw-master-admin-secret-key
```
### Key Lifecycle Rules
1. **Virtual API Keys (`sk-gw-...`)**: Scoped to specific Organizations and Teams, enforce sliding-window RPM/TPM rate limits, adhere to model allowlists, and cap monthly micro-dollar expenditure.
2. **Master Admin Key**: Bypasses rate limits and tenant isolation; used exclusively by automated control planes, CI/CD provisioning pipelines, and administrative operators.
3. **Provider Credentials**: Kept strictly inside GoRoute's AES-256-GCM encrypted vault; never exposed to callers or sent across client network boundaries.
---
## Standard Response Telemetry Headers
Every inference request executed by GoRoute returns real-time tracing, routing, and FinOps attribution headers:
| Header | Description | Example |
| :--- | :--- | :--- |
| `X-Gateway-Request-Id` | Unique UUID assigned to the request lifecycle | `req_991efda4-7fab-4b03` |
| `X-Trace-Id` | W3C / OpenTelemetry distributed trace identifier | `bad98e7deea34d7c810a963d9ed2c4ef` |
| `X-Gateway-Provider` | The winning upstream provider selected for execution | `gemini`, `anthropic`, `openai` |
| `X-Gateway-Model` | The physical upstream model that executed the inference | `gemini-2.5-flash`, `claude-3-5-sonnet-20241022` |
| `X-Gateway-Cost-Micros` | Exact micro-dollar spend incurred ($0.000001 USD) | `45` ($0.000045 USD) |
| `X-Gateway-Routing-Time` | Total Stage 1 + Stage 2 routing overhead | `14 µs` |
| `X-Gateway-Cache` | Exact-match cache evaluation status | `HIT` or `MISS` |
| `Retry-After` | Cooldown period sent on HTTP 429 rate limit errors | `45` (seconds) |
---
## Interactive Management via Operator Web UI
In addition to programmatic REST and gRPC interfaces, GoRoute serves an embedded single-page visual administration console directly from the binary at:
```text
http://localhost:8080/ui/
```
Use the Operator UI to issue virtual keys, inspect live deployment latencies, simulate security guardrails, test streaming chat completions, and monitor real-time FinOps ledgers.
---
## Next Steps
- [Gateway Data Plane API Reference](https://goroute-ai.github.io/docs/api/gateway-api)
- [Admin Control Plane API Reference](https://goroute-ai.github.io/docs/api/admin-api)
- [Model Context Protocol (MCP) API Reference](https://goroute-ai.github.io/docs/api/mcp-api)
- [Agent-to-Agent (A2A) Protocol API Reference](https://goroute-ai.github.io/docs/api/a2a-api)
- [Operator UI Administration Guide](https://goroute-ai.github.io/docs/deploy/operator-ui)
---
## Gateway Data Plane API Reference
URL: https://goroute-ai.github.io/docs/api/gateway-api
Markdown: https://goroute-ai.github.io/docs/api/gateway-api.md
Section: API Reference > API reference
# Gateway Data Plane API Reference
GoRoute AI Gateway exposes an OpenAI-compatible data plane supporting **24 primary `/v1` paths** across **73 `ProviderType` profiles** (plus `CustomOpenAICompatible`). All routes execute with sub-2ms gateway overhead, automated circuit breaking, cross-provider fallbacks, and real-time micro-dollar accounting.
> [!NOTE]
> All data plane routes accept standard OpenAI SDKs (Python, TypeScript, Go) and raw HTTP clients. Set your client `base_url` to `http://localhost:8080/v1` and supply a Virtual API Key (`sk-gw-...`) via the `Authorization` header.
---
## Authentication & Standard Telemetry Headers
Every data plane request requires a tenant-scoped Virtual API Key:
```http
Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e
```
### Standard Gateway Response Headers
Every response returned by GoRoute includes rich tracing, routing, and FinOps telemetry:
| Header | Description | Example |
| :--- | :--- | :--- |
| `X-Gateway-Request-Id` | Unique UUID assigned to the request lifecycle | `req_991efda4-7fab-4b03` |
| `X-Trace-Id` | W3C / OpenTelemetry distributed trace identifier | `bad98e7deea34d7c810a963d9ed2c4ef` |
| `X-Gateway-Provider` | The winning upstream provider selected for execution | `gemini`, `anthropic`, `openai` |
| `X-Gateway-Model` | The physical upstream model that processed the request | `gemini-2.5-flash`, `claude-3-5-sonnet-20241022` |
| `X-Gateway-Cost-Micros` | Exact micro-dollar spend incurred ($0.000001 USD) | `45` ($0.000045 USD) |
| `X-Gateway-Routing-Time` | Gateway Stage 1 + Stage 2 routing overhead | `14 µs` |
| `X-Gateway-Cache` | Exact-match cache evaluation status | `HIT` or `MISS` |
| `Retry-After` | Cooldown period sent on HTTP 429 rate limit errors | `45` (seconds) |
---
## Direct Dynamic `provider/model` Routing
In addition to logical aliases (`fast`, `reasoning`), GoRoute allows clients to bypass model groups and target upstream models directly using `provider/model` syntax (e.g., `anthropic/claude-3-5-sonnet-20241022`, `cerebras/llama-3.3-70b`, `bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0`, `vertex/gemini-2.5-pro`):
- **Layer 2 Provider Subscription Check**: The caller's organization must have an active subscription for the target provider.
- **Layer 3 Model Policy Check**: The requested model must not be disabled by the organization's model availability policy.
- **Tenant Credential Isolation**: The organization must have an active credential for the target provider stored in the vault. Host environment variables are never leaked across tenant boundaries.
- **Catalog Provisioning**: The gateway may provision model records from its catalog when needed. Catalog contents and discovery behavior depend on the gateway build and configuration; a provider-side model ID is not automatically available to every tenant or key.
- **Fail-Closed Boundary**: If repository queries fail or the model is unconfigured/unauthorized, GoRoute fails closed and returns `HTTP 404 model_group_not_found` (explicit model requests never silently fall back to unrelated tenant model groups).
---
## Summary of All 24 Primary `/v1` Data Plane Paths
| HTTP Method | Route | Description |
| :--- | :--- | :--- |
| `POST` | `/v1/chat/completions` | Multi-provider synchronous & SSE streaming chat inference |
| `POST` | `/v1/completions` | Legacy prompt text completions |
| `POST` | `/v1/embeddings` | Dense vector embeddings generation |
| `GET` | `/v1/models` | Tenant-filtered model catalogue and pricing inventory |
| `POST` | `/v1/rerank` | Document semantic reranking via Cohere v2 / Jina AI |
| `POST` | `/v1/moderations` | Automated content moderation and safety scoring |
| `POST` | `/v1/messages` | Native Anthropic Messages API ingress |
| `POST` | `/v1/messages/count_tokens` | Anthropic-compatible preflight token counting |
| `POST` | `/v1/tokenize` | In-process fast heuristic token estimation |
| `POST` | `/v1/audio/transcriptions` | Speech-to-text transcription with Whisper disk spooling |
| `POST` | `/v1/audio/translations` | Speech-to-text translation to English |
| `POST` | `/v1/audio/speech` | Text-to-speech synthesis (TTS) |
| `POST` | `/v1/images/generations` | Image generation (DALL-E 2/3, Imagen) |
| `POST` | `/v1/images/edits` | Image inpainting with mask uploads |
| `POST` | `/v1/images/variations` | Image variation generation |
| `POST` | `/v1/files` | Multipart file upload with SHA-256 deduplication |
| `GET` | `/v1/files` | List tenant files |
| `GET` | `/v1/files/{file_id}` | Retrieve file metadata |
| `GET` | `/v1/files/{file_id}/content`| Download file stream |
| `DELETE`| `/v1/files/{file_id}` | Delete stateful tenant file |
| `POST` | `/v1/batches` | Create asynchronous 24-hr batch job (50% discount) |
| `GET` | `/v1/batches` | List tenant batch jobs |
| `GET` | `/v1/batches/{batch_id}` | Inspect batch processing state and item counts |
| `POST` | `/v1/batches/{batch_id}/cancel` | Abort in-flight batch execution |
| `POST` | `/v1/responses` | OpenAI Responses API translation bridge |
| `GET` | `/v1/realtime` | Full-duplex OpenAI Realtime WebSocket session |
| `POST` | `/v1/guardrails/apply` | Standalone Pipeline v2 guardrail evaluation without upstream LLM call |
---
## 1. Chat Completions (`POST /v1/chat/completions`)
Standardized endpoint supporting Google Gemini, Anthropic Claude, OpenAI, Mistral, and local Ollama models in synchronous JSON or real-time Server-Sent Events (SSE) streaming mode.
### Synchronous Request
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "fast",
"messages": [
{"role": "system", "content": "You are a concise Go systems engineer."},
{"role": "user", "content": "How do goroutines manage stack growth?"}
],
"temperature": 0.7,
"max_tokens": 512,
"stream": false
}'
```
### Synchronous Response (`HTTP 200 OK`)
```json
{
"id": "chatcmpl-req_991efda4-7fab-4b03",
"object": "chat.completion",
"created": 1731081600,
"model": "gemini-2.5-flash",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Goroutines start with a small stack (typically 2KB in modern Go). When stack space is exhausted, the runtime allocates a contiguous block twice the size and copies the existing stack over."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 24,
"completion_tokens": 42,
"total_tokens": 66
}
}
```
### Real-Time SSE Stream (`stream: true`)
When `stream: true` is set, GoRoute initiates a zero-buffer Server-Sent Events stream:
```http
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
data: {"id":"chatcmpl-req_1","object":"chat.completion.chunk","created":1731081600,"model":"gemini-2.5-flash","choices":[{"index":0,"delta":{"role":"assistant","content":"Goroutines"},"finish_reason":null}]}
data: {"id":"chatcmpl-req_1","object":"chat.completion.chunk","created":1731081600,"model":"gemini-2.5-flash","choices":[{"index":0,"delta":{"content":" start"},"finish_reason":null}]}
data: {"id":"chatcmpl-req_1","object":"chat.completion.chunk","created":1731081600,"model":"gemini-2.5-flash","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]
```
### Extended Reasoning & Parameter Normalization
GoRoute transparently normalizes requests targeting frontier reasoning models:
- **Reasoning Effort (`reasoning_effort`: "low" | "medium" | "high")**:
- **OpenAI Reasoning (`o1`, `o3`, `o3-mini`)**: Converts `role: "system"` to `role: "developer"`, translates `max_tokens` to `max_completion_tokens`, and automatically strips unsupported sampling parameters (`temperature`, `top_p`).
- **Anthropic Claude**: Enables extended thinking with token budgets (`1024`, `4096`, `16384`) and omits `temperature`/`top_p` as mandated by Anthropic.
- **Google Gemini**: Sets `thinkingConfig.thinkingBudget` for Gemini 2.5 thinking models.
- **Anthropic Role Alternation on Tools**: When returning multiple parallel tool outputs (`role: "tool"`), GoRoute automatically consolidates consecutive tool results into a single `user` message with multiple `tool_result` content blocks to satisfy Anthropic's strict alternating turn protocol.
---
## 2. Legacy Text Completions (`POST /v1/completions`)
OpenAI-compatible prompt completion endpoint supporting single string prompts or string arrays.
### cURL Request
```bash
curl -X POST http://localhost:8080/v1/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-3.5-turbo-instruct",
"prompt": "Write a haiku about Go channels",
"max_tokens": 64,
"temperature": 0.5
}'
```
### Response (`HTTP 200 OK`)
```json
{
"id": "cmpl-req_7721a9c4",
"object": "text_completion",
"created": 1731081600,
"model": "gpt-3.5-turbo-instruct",
"choices": [
{
"text": "\nMessages flow like streams,\nGoroutines pass without lock,\nSync in harmony.",
"index": 0,
"logprobs": null,
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 7,
"completion_tokens": 19,
"total_tokens": 26
}
}
```
---
## 3. Vector Embeddings (`POST /v1/embeddings`)
Generates dense vector embeddings across single strings or string arrays, with optional encoding formats (`float` or `base64`) and dimension truncation.
### cURL Request
```bash
curl -X POST http://localhost:8080/v1/embeddings \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "text-embedding-3-small",
"input": ["GoRoute AI Gateway is ultra-fast", "Pure Go architecture"],
"encoding_format": "float"
}'
```
### Response (`HTTP 200 OK`)
```json
{
"object": "list",
"data": [
{
"object": "embedding",
"index": 0,
"embedding": [0.0123, -0.0456, 0.0891, -0.0012]
},
{
"object": "embedding",
"index": 1,
"embedding": [0.0341, -0.0128, 0.0712, 0.0054]
}
],
"model": "text-embedding-3-small",
"usage": {
"prompt_tokens": 14,
"total_tokens": 14
}
}
```
---
## 4. Model Catalogue Inventory (`GET /v1/models`)
Returns the tenant-filtered inventory of models accessible to the caller's Virtual API Key, evaluated against the 4-layer model access control hierarchy. Use this endpoint to check which model IDs are exposed to your caller. Provider catalogs change over time, and this response is deployment-specific; it is not a universal list of all models offered by upstream providers. The response may include capability and pricing metadata where available.
### cURL Request
```bash
curl -X GET http://localhost:8080/v1/models \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
### Illustrative response (`HTTP 200 OK`)
The model and pricing values below are sample data, not a current provider catalog or an availability guarantee.
```json
{
"object": "list",
"data": [
{
"id": "openai/gpt-4o",
"object": "model",
"created": 1731081600,
"owned_by": "openai",
"display_name": "OpenAI GPT-4o",
"max_input_tokens": 128000,
"max_output_tokens": 16384,
"capabilities": {
"tools": "supported",
"streaming": "supported",
"vision": "supported",
"reasoning": "unsupported",
"structured_outputs": "supported",
"embeddings": "unsupported"
},
"supports_streaming": true,
"supports_tools": true,
"supports_vision": true,
"supports_reasoning": false,
"supports_embeddings": false,
"input_cost_micros_per_1m": 2500000,
"output_cost_micros_per_1m": 10000000,
"cached_input_cost_micros_per_1m": 1250000,
"pricing_status": "verified",
"pricing_provenance_source": "official-vendor-pricing-2025",
"pricing_effective_date": "2025-05-01",
"lifecycle_status": "active"
}
]
}
```
---
## 5. Document Reranking (`POST /v1/rerank`)
Calculates semantic relevance scores between a search query and a list of text documents. Integrates with Cohere v2 API and Jina AI rerankers. Aliased at `/rerank` and `/v2/rerank`.
### cURL Request
```bash
curl -X POST http://localhost:8080/v1/rerank \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "rerank-v3.5",
"query": "What is the capital of France?",
"documents": [
"Carson City is the capital city of Nevada.",
"The capital of France is Paris.",
"Tokyo is the capital of Japan."
],
"top_n": 2,
"return_documents": true
}'
```
### Response (`HTTP 200 OK`)
```json
{
"id": "rerank-req_4ebe7b01",
"model": "rerank-v3.5",
"provider": "cohere",
"results": [
{
"index": 1,
"relevance_score": 0.9984,
"document": { "text": "The capital of France is Paris." }
},
{
"index": 0,
"relevance_score": 0.1245,
"document": { "text": "Carson City is the capital city of Nevada." }
}
],
"usage": {
"search_units": 1,
"total_tokens": 42
}
}
```
---
## 6. Content Moderation (`POST /v1/moderations`)
Evaluates text against safety categories using upstream moderation models (e.g. OpenAI `omni-moderation-latest`). Aliased at `/moderations`.
### cURL Request
```bash
curl -X POST http://localhost:8080/v1/moderations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "omni-moderation-latest",
"input": ["GoRoute AI Gateway is safe and reliable."]
}'
```
### Response (`HTTP 200 OK`)
```json
{
"id": "modr-req_865bcf43",
"model": "omni-moderation-latest",
"results": [
{
"flagged": false,
"categories": {
"hate": false,
"violence": false,
"self-harm": false,
"sexual": false
},
"category_scores": {
"hate": 0.00001,
"violence": 0.00002,
"self-harm": 0.00001,
"sexual": 0.00003
}
}
]
}
```
---
## 7. Anthropic Messages & Token Counting
Native Anthropic ingress bridges allowing Anthropic SDKs to route directly through GoRoute.
### Anthropic Messages (`POST /v1/messages`)
```bash
curl -X POST http://localhost:8080/v1/messages \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-3-5-sonnet-20241022",
"max_tokens": 1024,
"system": "You are a concise assistant.",
"messages": [
{"role": "user", "content": "Explain goroutines in 2 sentences."}
]
}'
```
### Anthropic Token Counting (`POST /v1/messages/count_tokens`)
Preflight token calculation conforming to Anthropic's count tokens schema:
```bash
curl -X POST http://localhost:8080/v1/messages/count_tokens \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "claude-3-5-sonnet-20241022",
"messages": [
{"role": "user", "content": "Explain goroutines in 2 sentences."}
]
}'
```
### Response (`HTTP 200 OK`)
```json
{
"input_tokens": 16
}
```
---
## 8. Fast Heuristic Tokenization (`POST /v1/tokenize`)
Zero-latency, in-process token counter approximating BPE without network calls. Aliased at `/tokenize`.
### cURL Request
```bash
curl -X POST http://localhost:8080/v1/tokenize \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"prompt": "GoRoute AI Gateway provides ultra-fast routing and FinOps accounting."
}'
```
### Response (`HTTP 200 OK`)
```json
{
"count": 13,
"model": "gpt-4o",
"profile": "cl100k_base",
"total_tokens": 13
}
```
---
## 9. Audio STT & TTS (`/v1/audio/*`)
Supports multipart audio uploads spooled to disk for zero-memory retryability, with micro-dollar FinOps billing based on integer duration seconds and character count.
### Audio Transcription (`POST /v1/audio/transcriptions`)
```bash
curl -X POST http://localhost:8080/v1/audio/transcriptions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@meeting_audio.mp3" \
-F "model=whisper-1" \
-F "language=en"
```
### Audio Translation (`POST /v1/audio/translations`)
Translates foreign audio files into English text:
```bash
curl -X POST http://localhost:8080/v1/audio/translations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@french_speech.mp3" \
-F "model=whisper-1"
```
### Audio Speech Synthesis (`POST /v1/audio/speech`)
Synthesizes text into streaming audio (formats: `mp3`, `opus`, `aac`, `flac`, `wav`):
```bash
curl -X POST http://localhost:8080/v1/audio/speech \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "tts-1",
"input": "GoRoute AI Gateway deployed in production.",
"voice": "alloy",
"response_format": "mp3"
}' --output speech.mp3
```
---
## 10. Image Generation, Edits & Variations (`/v1/images/*`)
Supports DALL-E 2, DALL-E 3, and Imagen models, returning `url` or `b64_json` payloads with exact micro-dollar pricing matrices keyed to quality (`standard` vs `hd`) and dimensions (`1024x1024`, `1024x1792`, `1792x1024`).
### Image Generation (`POST /v1/images/generations`)
```bash
curl -X POST http://localhost:8080/v1/images/generations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "dall-e-3",
"prompt": "Futuristic cloud architecture diagram rendered in neon blue",
"size": "1024x1024",
"quality": "standard",
"n": 1
}'
```
### Image Edits / Inpainting (`POST /v1/images/edits`)
```bash
curl -X POST http://localhost:8080/v1/images/edits \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "image=@base_image.png" \
-F "mask=@mask.png" \
-F "prompt": "Add a glowing golden trophy on the table" \
-F "model=dall-e-2"
```
### Image Variations (`POST /v1/images/variations`)
```bash
curl -X POST http://localhost:8080/v1/images/variations \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "image=@logo.png" \
-F "model=dall-e-2" \
-F "n=2"
```
---
## 11. Stateful Tenant Files (`/v1/files/*`)
Multi-tenant secure file storage with `0700` directory and `0600` file modes, automatic streaming SHA-256 deduplication indexing, and PostgreSQL metadata.
### Upload File (`POST /v1/files`)
```bash
curl -X POST http://localhost:8080/v1/files \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@batch_requests.jsonl" \
-F "purpose=batch"
```
Response (`HTTP 201 Created`):
```json
{
"id": "file-8f91a2bc-3d4e",
"object": "file",
"bytes": 1048576,
"created_at": 1731081600,
"filename": "batch_requests.jsonl",
"purpose": "batch",
"status": "uploaded"
}
```
### List Files (`GET /v1/files`)
```bash
curl http://localhost:8080/v1/files?purpose=batch \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
### Retrieve File Metadata (`GET /v1/files/{file_id}`)
```bash
curl http://localhost:8080/v1/files/file-8f91a2bc-3d4e \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
### Download File Stream (`GET /v1/files/{file_id}/content`)
```bash
curl http://localhost:8080/v1/files/file-8f91a2bc-3d4e/content \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
--output downloaded_batch.jsonl
```
### Delete File (`DELETE /v1/files/{file_id}`)
```bash
curl -X DELETE http://localhost:8080/v1/files/file-8f91a2bc-3d4e \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
---
## 12. Asynchronous Batches (`/v1/batches/*`)
Processes JSONL line items asynchronously within a 24-hour completion window for a 50% discount. State transitions: `validating` → `in_progress` → `completed` | `failed` | `cancelled`.
### Create Batch (`POST /v1/batches`)
```bash
curl -X POST http://localhost:8080/v1/batches \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file-8f91a2bc-3d4e",
"endpoint": "/v1/chat/completions",
"completion_window": "24h",
"metadata": { "job": "nightly_summaries" }
}'
```
Response (`HTTP 201 Created`):
```json
{
"id": "batch-3c91a0b1-4f2e",
"object": "batch",
"endpoint": "/v1/chat/completions",
"input_file_id": "file-8f91a2bc-3d4e",
"completion_window": "24h",
"status": "validating",
"created_at": 1731081600,
"request_counts": {
"total": 0,
"completed": 0,
"failed": 0
}
}
```
### Inspect Batch (`GET /v1/batches/{batch_id}`)
```bash
curl http://localhost:8080/v1/batches/batch-3c91a0b1-4f2e \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
### Cancel Batch (`POST /v1/batches/{batch_id}/cancel`)
```bash
curl -X POST http://localhost:8080/v1/batches/batch-3c91a0b1-4f2e/cancel \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
---
## 13. OpenAI Responses API (`POST /v1/responses`)
Stateless translation bridge conforming to the OpenAI Responses API schema (`input`, `instructions`), routing through GoRoute's 7-stage reliability pipeline.
### cURL Request
```bash
curl -X POST http://localhost:8080/v1/responses \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o",
"input": "Summarize quantum computing principles",
"instructions": "Be concise, use bullet points",
"stream": true
}'
```
### SSE Stream Format
Emits Responses events (`response.created`, `response.output_item.added`, `response.output_text.delta`, `response.output_text.done`, `response.output_item.done`, `response.completed`, `response.done`):
```http
event: response.created
data: {"response":{"id":"resp_0192a","status":"in_progress","model":"gpt-4o"}}
event: response.output_text.delta
data: {"output_index":0,"delta":"Quantum computing utilizes superposition and entanglement."}
event: response.completed
data: {"response":{"id":"resp_0192a","status":"completed"}}
event: response.done
data: {}
```
---
## 14. OpenAI Realtime WebSocket (`GET /v1/realtime`)
Full-duplex bidirectional WebSocket session handler supporting real-time speech and text streaming. Backed by `nhooyr.io/websocket` connection hijacking.
### Connecting via WebSocket
Clients connect to `ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview` passing authentication via Bearer header or subprotocol:
```javascript
const ws = new WebSocket(
'ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview',
['realtime', 'openai-insecure-api-key.sk-gw-prod-7f9a2b8c4d1e']
);
ws.onopen = () => {
ws.send(JSON.stringify({
type: 'response.create',
response: {
modalities: ['text'],
instructions: 'You are a real-time speech assistant.'
}
}));
};
```
---
## Next Steps
- Explore the [Admin Control Plane API](https://goroute-ai.github.io/docs/api/admin-api) for key provisioning and guardrails.
- Set up tool execution with the [MCP API](https://goroute-ai.github.io/docs/api/mcp-api).
- Connect multi-agent networks via the [A2A API](https://goroute-ai.github.io/docs/api/a2a-api).
---
## Admin Control Plane API Reference
URL: https://goroute-ai.github.io/docs/api/admin-api
Markdown: https://goroute-ai.github.io/docs/api/admin-api.md
Section: API Reference > API reference
# Admin Control Plane API Reference
The GoRoute Admin Control Plane (`/admin/v1/*`) provides unified APIs for tenant organization onboarding, multi-tenant RBAC, virtual key issuance, encrypted credential vault management, dynamic model routing, security guardrails, FinOps ledgers, and health probing.
> [!NOTE]
> All admin routes require authentication using the Master Admin Key (`GATEWAY_MASTER_ADMIN_KEY`):
> ```http
> Authorization: Bearer sk-gw-master-admin-secret-key
> ```
---
## Control Plane Groups Summary
| # | Group | Primary Endpoints | Core Function |
| :--- | :--- | :--- | :--- |
| 1 | **Organizations** | `POST/GET /admin/v1/organizations` | Tenant org creation and budget caps |
| 2 | **Subscriptions** | `POST/GET /admin/v1/organizations/{orgID}/subscriptions` | Layer 2 provider enablement |
| 3 | **Model Policies** | `POST/GET /admin/v1/organizations/{orgID}/models` | Layer 3 org model overrides and token ceilings |
| 4 | **Guardrails & Simulator** | `POST/GET /admin/v1/organizations/{orgID}/guardrails`, `/simulate`, `/guardrail-definitions`, `/guardrail-bindings`, `/tool-policies`, `/guardrail-usage` | PII/secret scanning, named registry (19 drivers, Starlark), hierarchical bindings, tool taint policies, and micro-dollar guardrail usage |
| 5 | **Teams** | `POST/GET /admin/v1/teams` | Departmental team isolation and model scopes |
| 6 | **Users** | `POST/GET /admin/v1/users` | User directory and role-based permissions |
| 7 | **Virtual API Keys** | `POST/GET /admin/v1/keys`, `POST .../revoke`, `DELETE` | Scoped API key issuance and instant revocation |
| 8 | **Credentials Vault** | `POST/GET /admin/v1/credentials`, `/validate`, `.../rotate` | AES-256-GCM encrypted provider secrets vault |
| 9 | **Model Groups & Deployments** | `POST/GET /admin/v1/model-groups`, `/deployments` | Virtual aliases, EWMA balancers, weight matrices |
| 10 | **Model Catalogue & Pricing** | `GET/POST /admin/v1/catalogue`, `/sync`, `/pricing` | Query and sync catalog records; manage pricing provenance |
| 11 | **Telemetry & Health Probes** | `GET /admin/v1/observability/metrics`, `/health/deployments`, `/cache/invalidate` | EWMA latencies, synthetic probes, cache purges |
| 12 | **FinOps & Audit Logs** | `GET /admin/v1/spend/summary`, `/spend/logs`, `/audit/logs` | Micro-dollar accounting ledgers and governance trails |
---
## 1. Organizations
Manages enterprise tenants, baseline micro-dollar budgets, and spend reset intervals.
### Create Organization (`POST /admin/v1/organizations`)
```bash
curl -X POST http://localhost:8080/admin/v1/organizations \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"id": "org_acme",
"name": "Acme Corporation",
"max_budget_micros": 500000000,
"budget_reset_period": "monthly"
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "org_acme",
"name": "Acme Corporation",
"max_budget_micros": 500000000,
"budget_reset_period": "monthly",
"is_active": true,
"created_at": "2026-11-10T12:00:00Z"
}
```
### List Organizations (`GET /admin/v1/organizations`)
```bash
curl http://localhost:8080/admin/v1/organizations \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"data": [
{
"id": "org_acme",
"name": "Acme Corporation",
"max_budget_micros": 500000000,
"budget_reset_period": "monthly",
"is_active": true
}
]
}
```
---
## 2. Organization Subscriptions (Layer 2 Access Control)
Controls which upstream cloud providers (OpenAI, Anthropic, Gemini, Mistral, Cohere) a tenant is authorized to invoke.
### Configure Provider Subscription (`POST /admin/v1/organizations/{orgID}/subscriptions`)
```bash
curl -X POST http://localhost:8080/admin/v1/organizations/org_acme/subscriptions \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"provider": "anthropic",
"is_enabled": true
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"org_id": "org_acme",
"provider": "anthropic",
"is_enabled": true,
"updated_at": "2026-11-10T12:05:00Z"
}
```
### Get Organization Subscriptions (`GET /admin/v1/organizations/{orgID}/subscriptions`)
```bash
curl http://localhost:8080/admin/v1/organizations/org_acme/subscriptions \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"data": [
{ "provider": "anthropic", "is_enabled": true },
{ "provider": "gemini", "is_enabled": true },
{ "provider": "openai", "is_enabled": false }
]
}
```
---
## 3. Organization Model Policies (Layer 3 Access Control)
Defines org-level model availability overrides and hard caps on output token generation.
### Set Model Policy (`POST /admin/v1/organizations/{orgID}/models`)
```bash
curl -X POST http://localhost:8080/admin/v1/organizations/org_acme/models \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"global_model_id": "anthropic/claude-sonnet-5-5",
"is_enabled": true,
"max_tokens_override": 4096
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"org_id": "org_acme",
"global_model_id": "anthropic/claude-sonnet-5-5",
"is_enabled": true,
"max_tokens_override": 4096,
"updated_at": "2026-11-10T12:10:00Z"
}
```
### List Model Policies (`GET /admin/v1/organizations/{orgID}/models`)
```bash
curl http://localhost:8080/admin/v1/organizations/org_acme/models \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
---
## 4. Security Guardrails & Simulator
Configures regex-based Personally Identifiable Information (PII) redaction, secret scanning (e.g. AWS secrets, GitHub tokens, OpenAI keys), and prompt injection filters.
### Set Guardrail Policy (`POST /admin/v1/organizations/{orgID}/guardrails`)
```bash
curl -X POST http://localhost:8080/admin/v1/organizations/org_acme/guardrails \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"enable_secret_detection": true,
"secret_action": "redact",
"enable_pii_detection": true,
"pii_action": "redact",
"custom_regex_patterns": ["ACME-[0-9]{6}"]
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"org_id": "org_acme",
"enable_secret_detection": true,
"secret_action": "redact",
"enable_pii_detection": true,
"pii_action": "redact",
"custom_regex_patterns": ["ACME-[0-9]{6}"]
}
```
### Retrieve Guardrail Policy (`GET /admin/v1/organizations/{orgID}/guardrails`)
```bash
curl http://localhost:8080/admin/v1/organizations/org_acme/guardrails \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
### Stateless Guardrails Simulator (`POST /admin/v1/organizations/{orgID}/guardrails/simulate`)
Tests safety rules against sample prompts without saving records or incurring upstream LLM token costs:
```bash
curl -X POST http://localhost:8080/admin/v1/organizations/org_acme/guardrails/simulate \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"prompt": "Here is my secret sk-proj-1234567890abcdef and SSN 000-11-2222 for ticket ACME-102938",
"policy": {
"enable_secret_detection": true,
"secret_action": "redact",
"enable_pii_detection": true,
"pii_action": "redact",
"custom_regex_patterns": ["ACME-[0-9]{6}"]
}
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"allowed": true,
"applied_action": "redact",
"violations": ["secret_detected", "pii_detected", "custom_pattern_matched"],
"redacted": true,
"redacted_prompt": "Here is my secret [REDACTED_SECRET] and SSN [REDACTED_SSN] for ticket [REDACTED_CUSTOM]"
}
```
---
## 5. Teams
Isolates departments or project teams within an organization, assigning dedicated budgets and model subsets.
### Create Team (`POST /admin/v1/teams`)
```bash
curl -X POST http://localhost:8080/admin/v1/teams \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"org_id": "org_acme",
"name": "core-platform",
"max_budget_micros": 100000000,
"allowed_models": ["fast", "reasoning", "anthropic/claude-sonnet-5-5"]
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "team_core_plat",
"org_id": "org_acme",
"name": "core-platform",
"max_budget_micros": 100000000,
"allowed_models": ["fast", "reasoning", "anthropic/claude-sonnet-5-5"],
"created_at": "2026-11-10T12:15:00Z"
}
```
### List Teams (`GET /admin/v1/teams?org_id={orgID}`)
```bash
curl "http://localhost:8080/admin/v1/teams?org_id=org_acme" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
---
## 6. Users
Manages human developers, service accounts, and enterprise roles (`admin`, `developer`, `finance_auditor`).
### Create User (`POST /admin/v1/users`)
```bash
curl -X POST http://localhost:8080/admin/v1/users \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"org_id": "org_acme",
"team_id": "team_core_plat",
"email": "engineer@acme.com",
"role": "developer"
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "usr_99182ab",
"org_id": "org_acme",
"team_id": "team_core_plat",
"email": "engineer@acme.com",
"role": "developer",
"created_at": "2026-11-10T12:20:00Z"
}
```
### List Users (`GET /admin/v1/users?org_id={orgID}`)
```bash
curl "http://localhost:8080/admin/v1/users?org_id=org_acme" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
---
## 7. Virtual API Keys (Layer 4 Access Control)
Issues high-entropy virtual keys (`sk-gw-...`) with budget caps, RPM/TPM sliding windows, and granular model allowlists.
### Issue Virtual Key (`POST /admin/v1/keys`)
```bash
curl -X POST http://localhost:8080/admin/v1/keys \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"org_id": "org_acme",
"team_id": "team_core_plat",
"name": "orders-service-prod",
"role": "service_account",
"max_budget_micros": 50000000,
"rate_limit_rpm": 1200,
"rate_limit_tpm": 600000,
"allowed_models": ["fast", "gemini-2.5-flash"],
"allowed_mcp_servers": ["github-tools"],
"allowed_a2a_agents": ["analyst-agent"]
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "key_0192ab4f",
"org_id": "org_acme",
"team_id": "team_core_plat",
"name": "orders-service-prod",
"key": "sk-gw-prod-7f9a2b8c4d1e2f3a",
"key_hash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"max_budget_micros": 50000000,
"rate_limit_rpm": 1200,
"rate_limit_tpm": 600000,
"allowed_models": ["fast", "gemini-2.5-flash"],
"created_at": "2026-11-10T12:25:00Z"
}
```
> [!CAUTION]
> The plain-text key `key` is returned only once at creation time. GoRoute stores only the SHA-256 digest `key_hash`.
### List Virtual Keys (`GET /admin/v1/keys?org_id={orgID}`)
```bash
curl "http://localhost:8080/admin/v1/keys?org_id=org_acme" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
### Revoke Virtual Key (`POST /admin/v1/keys/{keyID}/revoke` or `DELETE`)
Instantly purges key permissions from L1 in-memory caches and Redis keyspace:
```bash
curl -X POST http://localhost:8080/admin/v1/keys/key_0192ab4f/revoke \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"id": "key_0192ab4f",
"status": "revoked"
}
```
---
## 8. Provider Credentials Vault
Manages upstream provider API keys encrypted at rest with AES-256-GCM. Provides live preflight validation and zero-downtime key rotation.
### Create Credential (`POST /admin/v1/credentials`)
```bash
curl -X POST http://localhost:8080/admin/v1/credentials \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"org_id": "org_acme",
"provider": "openai",
"api_key": "sk-proj-prod-secret-api-key",
"base_url": "https://api.openai.com/v1"
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "cred_0198aa2b",
"org_id": "org_acme",
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"is_valid": true,
"created_at": "2026-11-10T12:30:00Z"
}
```
### List Credentials (`GET /admin/v1/credentials`)
```bash
curl http://localhost:8080/admin/v1/credentials \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
> [!NOTE]
> All returned credentials mask secrets (e.g. `sk-proj-***`).
### Preflight Credential Validation (`POST /admin/v1/credentials/validate`)
Validates raw credentials against the upstream vendor API *before* storing them in the vault:
```bash
curl -X POST http://localhost:8080/admin/v1/credentials/validate \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"provider": "openai",
"api_key": "sk-proj-prod-secret-api-key",
"base_url": "https://api.openai.com/v1"
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"valid": true,
"validation_status": "success",
"discovered_models": 34,
"latency_ms": 118
}
```
### Rotate Credential (`POST /admin/v1/credentials/{credID}/rotate`)
Atomically updates the secret key with zero connection interruption:
```bash
curl -X POST http://localhost:8080/admin/v1/credentials/cred_0198aa2b/rotate \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"api_key": "sk-proj-new-rotated-secret-key"
}'
```
### Validate Stored Credential (`POST /admin/v1/credentials/{credID}/validate`)
Tests an already-persisted vault credential:
```bash
curl -X POST http://localhost:8080/admin/v1/credentials/cred_0198aa2b/validate \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
---
## 9. Model Groups & Deployments
Configures logical routing groups (aliases) and physical model deployment matrices.
### Create Model Group (`POST /admin/v1/model-groups`)
```bash
curl -X POST http://localhost:8080/admin/v1/model-groups \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "fast",
"routing_strategy": "weighted-shuffle",
"fallback_group_ids": ["reasoning"]
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "mg_fast_019",
"name": "fast",
"routing_strategy": "weighted-shuffle",
"fallback_group_ids": ["reasoning"]
}
```
### Create Deployment (`POST /admin/v1/deployments`)
Adds a model deployment to a group with priority, weight, and rate limit capacities:
```bash
curl -X POST http://localhost:8080/admin/v1/deployments \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"model_group_id": "mg_fast_019",
"global_model_id": "gemini/gemini-2.5-flash",
"credential_id": "cred_gemini_vault",
"priority": 1,
"weight": 70,
"rpm_capacity": 2000,
"tpm_capacity": 2000000
}'
```
#### Response (`HTTP 201 Created`)
```json
{
"id": "dep_0192cd",
"model_group_id": "mg_fast_019",
"global_model_id": "gemini/gemini-2.5-flash",
"priority": 1,
"weight": 70,
"health_state": "CLOSED"
}
```
### List Deployments (`GET /admin/v1/deployments?model_group_id={groupID}`)
```bash
curl "http://localhost:8080/admin/v1/deployments?model_group_id=mg_fast_019" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
---
## 10. Model Catalogue & Pricing Provenance
Queries and synchronizes model catalog records and manages pricing provenance. Catalog contents can change as provider offerings and discovery settings change; a catalog entry does not by itself guarantee that a model is available to every tenant or virtual key.
### Query Catalogue (`GET /admin/v1/catalogue?provider={provider}&search={query}`)
```bash
curl "http://localhost:8080/admin/v1/catalogue?provider=anthropic&search=claude" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
### Import Embedded Models (`POST /admin/v1/catalogue/import?provider={provider}`)
```bash
curl -X POST "http://localhost:8080/admin/v1/catalogue/import?provider=gemini" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
### Update Model Pricing (`POST /admin/v1/catalogue/pricing`)
Promotes pricing status to `"verified"` with contract provenance:
```bash
curl -X POST http://localhost:8080/admin/v1/catalogue/pricing \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"global_model_id": "gemini/gemini-2.5-flash",
"input_cost_micros_per_1m": 150000,
"output_cost_micros_per_1m": 600000,
"cached_input_cost_micros_per_1m": 37500,
"pricing_status": "verified",
"pricing_provenance_source": "official-vendor-pricing-2025",
"pricing_effective_date": "2025-11-01"
}'
```
### Pricing Revision Audit Trail (`GET /admin/v1/catalogue/pricing?global_model_id={modelID}`)
```bash
curl "http://localhost:8080/admin/v1/catalogue/pricing?global_model_id=gemini/gemini-2.5-flash" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
---
## 11. Telemetry, Health Probes & Cache Invalidation
Monitors runtime latency, executes synthetic probes across deployments, and purges response or virtual key caches.
### Metrics Snapshot (`GET /admin/v1/observability/metrics`)
Returns real-time JSON metrics with latency percentiles and cache hit ratios:
```bash
curl http://localhost:8080/admin/v1/observability/metrics \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"total_requests": 142850,
"latency_p50_ms": 1.45,
"latency_p95_ms": 3.12,
"latency_p99_ms": 4.88,
"cache_hit_ratio": 0.428,
"active_circuit_breakers": {
"closed": 12,
"half_open": 1,
"open": 0
}
}
```
### Deployment Health States (`GET /admin/v1/health/deployments`)
```bash
curl http://localhost:8080/admin/v1/health/deployments \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
### Trigger Synthetic Health Probes (`POST /admin/v1/health/probe`)
```bash
curl -X POST http://localhost:8080/admin/v1/health/probe \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
### Cache Invalidation (`POST /admin/v1/cache/invalidate`)
Purges cached virtual keys or exact-match response cache keys:
```bash
curl -X POST http://localhost:8080/admin/v1/cache/invalidate \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{
"org_id": "org_acme",
"key_id": "key_0192ab4f"
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"status": "cache_invalidated",
"purged_entries": 1
}
```
---
## 12. FinOps Spend & Governance Audit Logs
Access real-time spend analytics and immutable regulatory audit logs.
### Spend Summary (`GET /admin/v1/spend/summary?org_id={orgID}&team_id={teamID}`)
```bash
curl "http://localhost:8080/admin/v1/spend/summary?org_id=org_acme" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"org_id": "org_acme",
"total_spend_micros": 48291000,
"budget_max_micros": 500000000,
"budget_remaining_micros": 451709000,
"spend_by_model": {
"gemini/gemini-2.5-flash": 12450000,
"anthropic/claude-sonnet-5-5": 35841000
}
}
```
### Spend Logs (`GET /admin/v1/spend/logs?org_id={orgID}&limit={limit}&offset={offset}`)
```bash
curl "http://localhost:8080/admin/v1/spend/logs?org_id=org_acme&limit=10" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"data": [
{
"request_id": "req_991efda4-7fab-4b03",
"trace_id": "bad98e7deea34d7c810a963d9ed2c4ef",
"model": "gemini-2.5-flash",
"provider": "gemini",
"cost_micros": 45,
"prompt_tokens": 24,
"completion_tokens": 42,
"cached_tokens": 0,
"created_at": "2026-11-10T12:40:00Z"
}
],
"total": 1
}
```
### Governance Audit Logs (`GET /admin/v1/audit/logs?org_id={orgID}&limit={limit}`)
```bash
curl "http://localhost:8080/admin/v1/audit/logs?org_id=org_acme&limit=10" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
```
#### Response (`HTTP 200 OK`)
```json
{
"data": [
{
"actor_id": "admin",
"action": "virtual_key.issued",
"resource": "virtual_key",
"resource_id": "key_0192ab4f",
"details": { "name": "orders-service-prod", "rpm_limit": 1200 },
"created_at": "2026-11-10T12:25:00Z"
}
]
}
```
---
## Next Steps
- Explore the [Operator Web UI Guide](https://goroute-ai.github.io/docs/deploy/operator-ui) to manage these resources visually.
- Review [Virtual Keys & FinOps](https://goroute-ai.github.io/docs/governance/virtual-keys) for enterprise governance patterns.
- Read [Exact-Match Caching](https://goroute-ai.github.io/docs/gateway/caching) to optimize gateway hit ratios.
---
## Model Context Protocol (MCP) API Reference
URL: https://goroute-ai.github.io/docs/api/mcp-api
Markdown: https://goroute-ai.github.io/docs/api/mcp-api.md
Section: API Reference > API reference
# Model Context Protocol (MCP) API Reference
GoRoute AI Gateway implements the official **MCP 2026-07-28** stateless Streamable HTTP specification with backward compatibility for **MCP 2025-11-25**. It functions as an enterprise Virtual Hub aggregating distributed tool servers, validating schemas, enforcing Virtual Key scopes, and eliminating prompt token overhead with progressive discovery.
> [!NOTE]
> All MCP routes authenticate using the standard Bearer header with a tenant-scoped Virtual API Key:
> ```http
> Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e
> ```
---
## MCP Route Topology
| Protocol Method | Route | Description |
| :--- | :--- | :--- |
| `POST` | `/mcp` | Virtual Hub multi-server aggregator (`tools/list`, `tools/call`) |
| `POST` | `/mcp/{serverSlug}` | Scoped execution targeted at a single upstream MCP server |
| `POST` | `/mcp/toolsets/{toolsetSlug}` | Execution routed through a curated, vetted toolset |
| `GET` | `/mcp/subscriptions/listen` | Client SSE stream for dynamic catalog invalidation events |
| `GET` | `/mcp-rest/tools/list` | Plain REST list of aggregated tools without JSON-RPC framing |
| `POST` | `/mcp-rest/tools/call` | Plain REST tool execution with direct JSON payloads |
| `GET` | `/v1/goroute/mcp/registry` | Progressive discovery feed issuing 256-bit tool handles |
---
## 1. Virtual Hub Multi-Server Aggregator (`POST /mcp`)
Aggregates all registered, active, and tenant-authorized MCP servers into a single unified JSON-RPC 2.0 endpoint. Tool names are automatically namespaced (`__`) to prevent collisions.
### Headers Required
```http
Content-Type: application/json
Accept: application/json, text/event-stream
Origin: http://localhost:8080
Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e
```
### Tool Discovery (`tools/list`)
```bash
curl -X POST http://localhost:8080/mcp \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "tools/list",
"params": {}
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"tools": [
{
"name": "github__create_issue",
"description": "Create an issue in a GitHub repository",
"inputSchema": {
"type": "object",
"properties": {
"owner": { "type": "string" },
"repo": { "type": "string" },
"title": { "type": "string" },
"body": { "type": "string" }
},
"required": ["owner", "repo", "title"]
}
},
{
"name": "fin_tools__calculate_dcf",
"description": "Calculate discounted cash flow with high precision",
"inputSchema": {
"type": "object",
"properties": {
"discount_rate": { "type": "number" },
"cash_flows": { "type": "array", "items": { "type": "number" } }
},
"required": ["discount_rate", "cash_flows"]
}
}
],
"_meta": {
"goroute.ai/server_outcomes": {
"github": "success",
"fin-tools": "success"
}
}
}
}
```
### Tool Execution via Virtual Hub (`tools/call`)
```bash
curl -X POST http://localhost:8080/mcp \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 2,
"method": "tools/call",
"params": {
"name": "github__create_issue",
"arguments": {
"owner": "goroute-ai",
"repo": "ai-gateway-go",
"title": "Bug: fix connection pooling",
"body": "Detailed description of the issue."
}
}
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"jsonrpc": "2.0",
"id": 2,
"result": {
"content": [
{
"type": "text",
"text": "Issue #42 created successfully at https://github.com/goroute-ai/ai-gateway-go/issues/42"
}
],
"isError": false
}
}
```
---
## 2. Scoped Server Execution (`POST /mcp/{serverSlug}`)
Routes JSON-RPC 2.0 requests exclusively to the upstream server matching `{serverSlug}`. Tool names do not require namespace prefixes in scoped mode.
GoRoute preserves arbitrary decimal precision for financial and scientific tools using Go's `big.Rat` and `json.Number` without IEEE-754 floating point distortion.
```bash
curl -X POST http://localhost:8080/mcp/fin-tools \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 10,
"method": "tools/call",
"params": {
"name": "calculate_dcf",
"arguments": {
"discount_rate": 0.08251294819284719,
"cash_flows": [1000000, 1500000, 2000000]
}
}
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"jsonrpc": "2.0",
"id": 10,
"result": {
"content": [
{
"type": "text",
"text": "Net Present Value: 3824192.48291048192"
}
],
"isError": false
}
}
```
---
## 3. Curated Toolset Execution (`POST /mcp/toolsets/{toolsetSlug}`)
Executes tool calls through a predefined, vetted subset of pinned tools spanning multiple backend servers. Useful for creating role-restricted environments (e.g. `read-only-analyst`, `devops-tier-1`).
```bash
curl -X POST http://localhost:8080/mcp/toolsets/analyst-tools \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 20,
"method": "tools/list",
"params": {}
}'
```
---
## 4. Plain MCP REST Bridge (`/mcp-rest/*`)
Standard RESTful endpoints designed for traditional HTTP clients, SDKs, and webhooks that interact without JSON-RPC 2.0 framing.
### List Accessible Tools (`GET /mcp-rest/tools/list`)
#### Query Parameters
- `server_slug` *(optional)*: Filter tools registered under a specific upstream server.
```bash
curl "http://localhost:8080/mcp-rest/tools/list?server_slug=fin-tools" \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
#### Response (`HTTP 200 OK`)
```json
{
"tools": [
{
"id": "tool_019248ab-3c4d-5e6f",
"server_id": "mcp_srv_771829a",
"server_slug": "fin-tools",
"name": "calculate_dcf",
"description": "Calculate discounted cash flow with high precision",
"input_schema": {
"type": "object",
"properties": {
"discount_rate": { "type": "number" },
"cash_flows": { "type": "array" }
},
"required": ["discount_rate", "cash_flows"]
},
"cost_per_call_micros": 500,
"is_active": true
}
]
}
```
### Execute Tool Call (`POST /mcp-rest/tools/call`)
```bash
curl -X POST http://localhost:8080/mcp-rest/tools/call \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"server_slug": "fin-tools",
"tool_name": "calculate_dcf",
"arguments": {
"discount_rate": 0.08,
"cash_flows": [500000, 750000, 1000000]
}
}'
```
#### Response (`HTTP 200 OK`)
```json
{
"content": [
{
"type": "text",
"text": "Net Present Value: 1851240.22"
}
],
"is_error": false
}
```
#### Error Handling
- `HTTP 400 AmbiguousToolName`: Returned when `tool_name` matches multiple active servers and `server_slug` was omitted. Contains a `matching_servers` array.
- `HTTP 403 mcp_server_forbidden`: Caller's Virtual API Key is not authorized for the requested server or toolset.
---
## 5. Progressive Tool Discovery & Public Registry (`GET /v1/goroute/mcp/registry`)
To prevent token exhaustion when exposing hundreds of tools to LLM agent loops, GoRoute issues compact 256-bit cryptographic tool handles (`tool_hdl_<64 hex chars>`).
### Registry Catalog Feed
```bash
curl http://localhost:8080/v1/goroute/mcp/registry \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
#### Response (`HTTP 200 OK`)
```json
{
"servers": [
{
"server_slug": "fin-tools",
"tools": [
{
"name": "calculate_dcf",
"description": "Calculates discounted cash flow valuation",
"tool_handle": "tool_hdl_a1b2c3d4e5f67890123456789abcdef0123456789abcdef0123456789abcdef0"
}
]
}
]
}
```
### Dynamic Schema Hydration Meta-Tools
When progressive discovery is active in an agent loop, models only receive tool handles and descriptions. When a model decides to invoke a tool, it calls built-in meta-tools:
1. **`search_tools`**: Keyword search over registered tool definitions:
```json
{ "query": "discounted cash flow", "limit": 5 }
```
2. **`get_tool_schema`**: Resolves full JSON Schema by handle:
```json
{ "tool_handle": "tool_hdl_a1b2c3d4e5f67890123456789abcdef0123456789abcdef0123456789abcdef0" }
```
3. **Execution via Handle**: Directly call `POST /mcp-rest/tools/call` using `tool_name: "tool_hdl_..."`. If an upstream schema mutates, stale handles fail-closed with HTTP 404/410.
---
## Next Steps
- Consult the [Agent-to-Agent (A2A) API Reference](https://goroute-ai.github.io/docs/api/a2a-api).
- Learn about [MCP Stdio Sandboxing](https://goroute-ai.github.io/docs/agents/mcp-stdio-sandboxing).
- Manage servers in the [Operator Web UI](https://goroute-ai.github.io/docs/deploy/operator-ui).
---
## Agent-to-Agent (A2A) Protocol API Reference
URL: https://goroute-ai.github.io/docs/api/a2a-api
Markdown: https://goroute-ai.github.io/docs/api/a2a-api.md
Section: API Reference > API reference
# Agent-to-Agent (A2A) Protocol API Reference
GoRoute AI Gateway implements the Linux Foundation **Agent-to-Agent (A2A v1.0)** protocol specification. It serves as an enterprise gateway for autonomous agent discovery, asynchronous multi-step task delegation, real-time event streaming, and cross-framework agent federation.
> [!NOTE]
> All A2A data plane endpoints require standard Bearer token authentication using a tenant Virtual API Key:
> ```http
> Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e
> ```
---
## A2A Endpoints Summary
| Protocol / Method | Endpoint | Description |
| :--- | :--- | :--- |
| `GET` | `/.well-known/agent-card.json` | RFC 8785 JCS canonicalized & RFC 7515 JWS signed Agent Card |
| `POST` | `/a2a/v1/agents/{agentSlug}` | Dual-binding message dispatch (JSON-RPC 2.0 & HTTP+JSON) |
| `GET` | `/a2a/v1/agents/{agentSlug}/tasks` | List tasks managed by the agent |
| `GET` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}` | Inspect state, iteration budget, and outputs for a task |
| `POST` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}:cancel` | Safely abort an in-flight task execution |
| `GET` | `/a2a/v1/agents/{agentSlug}/tasks/{taskID}:subscribe` | Subscribe-First SSE stream emitting monotonic `a2a-event`s |
| `gRPC` | `lf.a2a.v1.A2AService` | Native high-performance gRPC service |
| `POST` | `/v1/chat/completions` (`model: "a2a/{slug}"`) | OpenAI-compatible bridge for multi-agent workflows |
---
## 1. Cryptographic Agent Card Discovery (`GET /.well-known/agent-card.json`)
Exposes the gateway's canonical agent identity formatted with **RFC 8785 JSON Canonicalization Scheme (JCS)** and signed with **RFC 7515 JWS (`ES256`)**. External agents and registries use this card to verify capabilities, cryptographic public keys, and supported endpoints.
### cURL Request
```bash
curl http://localhost:8080/.well-known/agent-card.json
```
### Response Schema (`HTTP 200 OK`)
```json
{
"name": "GoRoute AI Gateway Agent Service",
"version": "1.0.0",
"description": "Enterprise Autonomous Routing, Aggregation & Inference Agent",
"supportedInterfaces": [
{
"url": "http://localhost:8080/a2a/v1/agents/analyst",
"protocolBinding": "HTTP+JSON"
},
{
"url": "http://localhost:8080/a2a/v1/agents/analyst",
"protocolBinding": "JSON-RPC-2.0"
},
{
"url": "grpc://localhost:9090",
"protocolBinding": "gRPC"
}
],
"capabilities": {
"streaming": true,
"asyncTasks": true,
"pushNotifications": true
},
"signatures": [
{
"header": {
"alg": "ES256",
"kid": "gw-a2a-signer-v1",
"typ": "JWT"
},
"signature": "MEUCIQDx41..."
}
]
}
```
---
## 2. Dual-Binding Agent Message Dispatch (`POST /a2a/v1/agents/{agentSlug}`)
Dispatches messages to upstream agents. GoRoute inspects the request body and transparently handles both **JSON-RPC 2.0** and **HTTP+JSON Protobuf REST** formats.
### Option A: JSON-RPC 2.0 Binding (`tasks/send`)
```bash
curl -X POST http://localhost:8080/a2a/v1/agents/risk-analyst \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": "req-a2a-001",
"method": "tasks/send",
"params": {
"message": {
"role": "ROLE_USER",
"parts": [
{ "text": "Perform portfolio Value-at-Risk assessment for Portfolio Q3." }
]
}
}
}'
```
#### JSON-RPC 2.0 Response (`HTTP 200 OK`)
```json
{
"jsonrpc": "2.0",
"id": "req-a2a-001",
"result": {
"role": "ROLE_AGENT",
"parts": [
{ "text": "Portfolio risk assessment completed: VaR (95% confidence) is 1.42%." }
],
"taskId": "task_a2a_5b91a7c0",
"state": "completed"
}
}
```
### Option B: HTTP+JSON Protobuf Binding
```bash
curl -X POST http://localhost:8080/a2a/v1/agents/risk-analyst \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"message": {
"role": "ROLE_USER",
"parts": [
{ "text": "Calculate VaR 99% for portfolio." }
]
},
"budget": {
"max_cost_micros": 5000000,
"max_iterations": 10
}
}'
```
---
## 3. Task Management Endpoints
Inspect, list, and control asynchronous agent execution lifecycles.
### Inspect Task Details (`GET /a2a/v1/agents/{agentSlug}/tasks/{taskID}`)
```bash
curl http://localhost:8080/a2a/v1/agents/risk-analyst/tasks/task_a2a_5b91a7c0 \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
#### Response (`HTTP 200 OK`)
```json
{
"id": "task_a2a_5b91a7c0",
"agent_slug": "risk-analyst",
"state": "working",
"created_at": "2026-11-10T12:00:00Z",
"updated_at": "2026-11-10T12:00:05Z",
"iterations_executed": 3,
"max_iterations": 10,
"cost_incurred_micros": 12500
}
```
### Cancel In-Flight Task (`POST /a2a/v1/agents/{agentSlug}/tasks/{taskID}:cancel`)
Safely terminates background execution and prevents further upstream model spend:
```bash
curl -X POST http://localhost:8080/a2a/v1/agents/risk-analyst/tasks/task_a2a_5b91a7c0:cancel \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"
```
---
## 4. Subscribe-First Real-Time SSE Task Streaming (`GET ...:subscribe`)
Clients connect to stream durable task events (status transitions, intermediate thoughts, artifact publications) in real-time. Emits `a2a-event` Server-Sent Events with strictly monotonic sequence numbers (`seq`).
```bash
curl -N http://localhost:8080/a2a/v1/agents/risk-analyst/tasks/task_a2a_5b91a7c0:subscribe \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Accept: text/event-stream"
```
### Event Stream Wire Format
```http
HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
event: a2a-event
data: {"seq":1,"taskId":"task_a2a_5b91a7c0","state":"working","timestamp":"2026-11-10T12:00:01Z"}
event: a2a-event
data: {"seq":2,"taskId":"task_a2a_5b91a7c0","state":"working","thought":"Fetching historical volatility for 50 tickers...","timestamp":"2026-11-10T12:00:02Z"}
event: a2a-event
data: {"seq":3,"taskId":"task_a2a_5b91a7c0","state":"completed","output":{"role":"ROLE_AGENT","parts":[{"text":"VaR calculation complete."}]},"timestamp":"2026-11-10T12:00:05Z"}
```
---
## 5. Native A2A gRPC Service (`lf.a2a.v1.A2AService`)
For latency-critical agent-to-agent communication, GoRoute exposes a native gRPC server implementing the official Linux Foundation Protobuf contract.
### Protobuf Service Definition
```protobuf
syntax = "proto3";
package lf.a2a.v1;
service A2AService {
rpc SendMessage(SendMessageRequest) returns (SendMessageResponse);
rpc GetTask(GetTaskRequest) returns (Task);
rpc ListTasks(ListTasksRequest) returns (ListTasksResponse);
rpc CancelTask(CancelTaskRequest) returns (Task);
rpc SubscribeToTask(SubscribeToTaskRequest) returns (stream StreamTaskResponse);
}
```
### Go gRPC Client Example
```go
package main
import (
"context"
"fmt"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
pb "github.com/goroute-ai/ai-gateway-go/internal/a2a/pb"
)
func main() {
conn, _ := grpc.Dial("localhost:9090", grpc.WithInsecure())
defer conn.Close()
client := pb.NewA2AServiceClient(conn)
md := metadata.Pairs("authorization", "Bearer sk-gw-prod-7f9a2b8c4d1e")
ctx := metadata.NewOutgoingContext(context.Background(), md)
resp, err := client.SendMessage(ctx, &pb.SendMessageRequest{
AgentSlug: "risk-analyst",
Message: &pb.Message{
Role: pb.MessageRole_ROLE_USER,
Parts: []*pb.Part{pb.NewTextPart("Run risk audit")},
},
})
if err != nil {
panic(err)
}
fmt.Printf("Task ID: %s, Response: %v\n", resp.TaskId, resp.Message)
}
```
---
## 6. A2A-to-OpenAI Compatibility Bridge
Allows standard OpenAI Python/TypeScript/Go SDKs to orchestrate multi-agent workflows without changing client libraries by specifying `model: "a2a/{agent_slug}"`.
### cURL Request via Chat Completions
```bash
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "a2a/risk-analyst",
"messages": [
{"role": "user", "content": "Compute Value-at-Risk for Portfolio 4"}
]
}'
```
#### Multi-Turn Resumption (`TASK_STATE_INPUT_REQUIRED`)
When an upstream agent requires clarifying input, GoRoute returns an assistant response with `finish_reason: "stop"` containing the agent's question. Callers resume execution simply by sending the next user turn back to `model: "a2a/{agent_slug}"`; GoRoute automatically reconciles conversational context with the active `taskId`.
---
## 7. Emergency Administrative Kill-Switch
Administrators can instantly disable an agent across all cluster nodes using the control plane API:
```bash
curl -X POST http://localhost:8080/admin/v1/a2a/agents/agent_0192ab/kill-switch \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{ "killed": true }'
```
When tripped, subsequent invocations immediately fail-closed returning `HTTP 503 agent_killed`.
---
## Next Steps
- Explore the [A2A Architectural Guide](https://goroute-ai.github.io/docs/agents/a2a-overview).
- Review [Model Context Protocol (MCP) APIs](https://goroute-ai.github.io/docs/api/mcp-api).
- Inspect [Audit Logs & FinOps Ledgers](https://goroute-ai.github.io/docs/api/admin-api).
---
## Go Examples & Task Cookbook
URL: https://goroute-ai.github.io/docs/examples/overview
Markdown: https://goroute-ai.github.io/docs/examples/overview.md
Section: Runnable Go Examples
# Go Examples & Task Cookbook
Use these runnable Go examples when you want to see how a task works in practice. Each guide connects the example to the related product documentation, so you can start with a small working path and then explore the relevant configuration.
---
## Find a Guide by Task
| Goal | Guide / Demo |
| :--- | :--- |
| **Start building** | [Go SDK Quickstart](https://goroute-ai.github.io/docs/start/quickstart-sdk) · [Standalone Gateway Quickstart](https://goroute-ai.github.io/docs/start/quickstart-gateway) · [Docker Compose](https://goroute-ai.github.io/docs/start/quickstart-docker) |
| **Connect Python/TS SDKs & Coding IDEs** | [Integrations Overview](https://goroute-ai.github.io/docs/integrations/overview) · [OpenAI SDKs](https://goroute-ai.github.io/docs/integrations/openai-sdks) · [Anthropic SDKs](https://goroute-ai.github.io/docs/integrations/anthropic-sdks) · [Claude Code](https://goroute-ai.github.io/docs/integrations/claude-code) · [Cursor & IDEs](https://goroute-ai.github.io/docs/integrations/cursor-and-ides) |
| **Configure routing, fallbacks & caching** | [Routing & Load Balancing](https://goroute-ai.github.io/docs/gateway/routing-and-load-balancing) · [Resiliency & Fallbacks](https://goroute-ai.github.io/docs/gateway/resiliency-and-fallbacks) · [Exact-Match Cache](https://goroute-ai.github.io/docs/gateway/caching) |
| **Enforce Virtual Keys, RBAC & Guardrails** | [Virtual Keys](https://goroute-ai.github.io/docs/governance/virtual-keys) · [RBAC & Vault](https://goroute-ai.github.io/docs/governance/rbac-and-vault) · [Guardrails](https://goroute-ai.github.io/docs/governance/guardrails) · [SSRF & Network Security](https://goroute-ai.github.io/docs/governance/network-and-ssrf) |
| **Track FinOps spend & rate limits** | [Micro-Dollar Ledger](https://goroute-ai.github.io/docs/finops/micro-dollar-ledger) · [Sliding-Window Rate Limits](https://goroute-ai.github.io/docs/finops/rate-limiting) · [Pre-Flight Tokenization](https://goroute-ai.github.io/docs/finops/preflight-tokenization) |
| **Monitor telemetry & troubleshoot** | [Observability & Health Probes](https://goroute-ai.github.io/docs/deploy/observability) · [OpenTelemetry & Prometheus](https://goroute-ai.github.io/docs/integrations/otel-and-prometheus) · [Operator Web UI](https://goroute-ai.github.io/docs/deploy/operator-ui) · [Troubleshooting](https://goroute-ai.github.io/docs/deploy/troubleshooting) |
| **Run MCP tool hubs & A2A agent networks** | [MCP Overview](https://goroute-ai.github.io/docs/agents/mcp-overview) · [A2A Overview](https://goroute-ai.github.io/docs/agents/a2a-overview) · [Autonomous Tool Loop](https://goroute-ai.github.io/docs/gateway/endpoints/autonomous-tool-loop) |
| **Deploy to production** | [Configuration](https://goroute-ai.github.io/docs/deploy/configuration) · [Deployment & Storage](https://goroute-ai.github.io/docs/deploy/deployment-and-storage) · [Kubernetes & Kustomize](https://goroute-ai.github.io/docs/deploy/kubernetes-kustomize) · [Production Checklist](https://goroute-ai.github.io/docs/deploy/production-checklist) |
---
## Running Any Demo
Run an example from the `examples/` module after its Go dependencies are available:
```bash
cd examples
go run ./hello_gateway
```
Most examples enable the built-in mock provider when credentials are missing, so they can demonstrate the request flow without contacting a model provider. Examples that require a local service, such as Ollama or an HTTP server, still need that service running. To test a live provider, export the matching key before running its example:
```bash
export OPENAI_API_KEY="sk-proj-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GEMINI_API_KEY="AIzaSy... or AQ.Ab8..."
```
The examples module declares its Go toolchain version in `examples/go.mod`. See each example's source for the provider, local service, or environment variables it needs.
---
## Examples by task
### Fundamentals
- [**Hello Gateway**](https://goroute-ai.github.io/docs/examples/hello-gateway): Minimal in-process gateway setup and synchronous chat.
- [**System Prompts**](https://goroute-ai.github.io/docs/examples/system-prompt): Persona steering using `gateway.WithSystem(...)`.
- [**Streaming Tokens**](https://goroute-ai.github.io/docs/examples/streaming-tokens): Zero-buffer SSE token streaming via `StreamChat`.
- [**Sampling & Limits**](https://goroute-ai.github.io/docs/examples/temperature-and-sampling): Deterministic outputs and minimal token capping.
- [**Structured JSON**](https://goroute-ai.github.io/docs/examples/json-mode-structured-output): Guaranteed machine-readable JSON object output.
- [**Multi-Turn Memory**](https://goroute-ai.github.io/docs/examples/multi-turn-conversation): Maintaining conversation history across multiple turns.
### Multi-Provider Integrations
- [**OpenAI Direct**](https://goroute-ai.github.io/docs/examples/openai-direct): Connecting to OpenAI with `openai/gpt-4o-mini`.
- [**Anthropic Direct**](https://goroute-ai.github.io/docs/examples/anthropic-direct): Connecting to Anthropic with `anthropic/claude-haiku-4-5-20251001`.
- [**Google Gemini Direct**](https://goroute-ai.github.io/docs/examples/gemini-direct): Connecting to Gemini with `gemini/gemini-2.5-flash-lite`.
- [**Local Ollama**](https://goroute-ai.github.io/docs/examples/local-ollama): Self-hosted offline inference on port 11434.
- [**Unified Model Aliases**](https://goroute-ai.github.io/docs/examples/unified-model-alias): Logical aliases decoupling callers from specific vendors.
### Tool Calling & Autonomous Agents
- [**Tool Definition**](https://goroute-ai.github.io/docs/examples/single-tool-definition): Registering tool schemas via Model Context Protocol (MCP).
- [**Tool Execution**](https://goroute-ai.github.io/docs/examples/tool-execution-callback): In-process tool execution and result returning.
- [**Multi-Tool Inventory**](https://goroute-ai.github.io/docs/examples/multi-tool-selection): Aggregating multiple tool servers into a single schema.
- [**Autonomous Agent Loop**](https://goroute-ai.github.io/docs/examples/autonomous-tool-loop): Multi-turn self-orchestrating loop with zero human intervention.
### Traffic Routing & Outage Resiliency
- [**Outage Fallback Chains**](https://goroute-ai.github.io/docs/examples/failover-fallback-chain): Automatic 503 outage recovery with secondary models.
- [**Round-Robin Balancer**](https://goroute-ai.github.io/docs/examples/round-robin-balancer): Uniform stateless load balancing.
- [**Weighted Shuffle Routing**](https://goroute-ai.github.io/docs/examples/weighted-shuffle-routing): Proportionate canary and cost-steered traffic splitting.
- [**Least-Latency EWMA**](https://goroute-ai.github.io/docs/examples/least-latency-ewma): Peak-EWMA routing favoring lowest latency endpoints.
- [**Circuit Breakers**](https://goroute-ai.github.io/docs/examples/circuit-breaker): Fast-failing broken backends to protect caller latency.
### Enterprise FinOps, Protocols & Microservices
- [**Exact-Match Cache**](https://goroute-ai.github.io/docs/examples/exact-match-cache): Sub-millisecond SHA-256 caching for identical prompts.
- [**Sliding-Window Rate Limits**](https://goroute-ai.github.io/docs/examples/sliding-window-rate-limiting): Strict RPM and TPM quota enforcement.
- [**Virtual Keys & Budgets**](https://goroute-ai.github.io/docs/examples/virtual-keys-and-budgets): Department keys, micro-dollar tracking, and revocation.
- [**MCP Virtual Hub**](https://goroute-ai.github.io/docs/examples/mcp-virtual-hub): Model Context Protocol Virtual Hub & REST bridge.
- [**A2A Agent Network**](https://goroute-ai.github.io/docs/examples/a2a-agent-network): Cryptographic Agent-to-Agent discovery and dispatch.
- [**Embedded Microservice**](https://goroute-ai.github.io/docs/examples/embedded-http-microservice): Production HTTP server with Kubernetes probes.
---
## Hello Gateway
URL: https://goroute-ai.github.io/docs/examples/hello-gateway
Markdown: https://goroute-ai.github.io/docs/examples/hello-gateway.md
Section: Runnable Go Examples > Get a request working
# Hello Gateway
In this guide, you will initialize the GoRoute AI Gateway in-process with a single model and execute a synchronous chat completion.
---
## Architecture Flow
```mermaid
sequenceDiagram
participant Caller as Go Application
participant GW as GoRoute AI Gateway (In-Process)
participant Provider as Upstream LLM / Mock Engine
Caller->>GW: 1. gateway.New(WithModelList(...))
Note over GW: Embedded server boots in < 2ms
Caller->>GW: 2. gw.Chat(ctx, "gpt-4o-mini", "Hello!")
GW->>Provider: 3. Forward Canonical Request
Provider-->>GW: 4. Return Completion
GW-->>Caller: 5. *gateway.ChatResult (Content, Cost, Latency)
```
---
## Runnable Go Code
```go title="examples/hello_gateway/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Hello Gateway (Minimal 20-Line In-Process Engine)")
// 1. Initialize embedded gateway with one model entry
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "gpt-4o-mini",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(), // Uses an in-process mock when the key is missing
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// 2. Execute chat completion
res, err := gw.Chat(ctx, "gpt-4o-mini", "Hello! What are you?")
if err != nil {
demoutil.Fail("Chat request failed", err)
}
demoutil.Info("Model", res.Model)
demoutil.Info("Provider", res.Provider)
demoutil.Info("Routing Latency (µs)", fmt.Sprintf("%d µs", res.RoutingMicros))
demoutil.Info("Cost (Micro-Dollars)", fmt.Sprintf("%d micros ($%.6f USD)", res.CostMicros, float64(res.CostMicros)/1e6))
demoutil.Success("Response", res.Content)
demoutil.Done()
}
```
---
## Execution Output
```text
================================================================================
Hello Gateway (Minimal 20-Line In-Process Engine)
================================================================================
• Model : gpt-4.1-mini
• Provider : openai
• Routing Latency (µs) : 1 µs
• Cost (Micro-Dollars) : 37 micros ($0.000037 USD)
✓ Response : OpenAI GPT-4.1-mini handled this request through the unified canonical gateway contract.
--------------------------------------------------------------------------------
```
---
## System Prompts
URL: https://goroute-ai.github.io/docs/examples/system-prompt
Markdown: https://goroute-ai.github.io/docs/examples/system-prompt.md
Section: Runnable Go Examples > Get a request working
# System Prompts
Learn how to steer the model's persona, tone, and constraints using `gateway.WithSystem(...)`. GoRoute automatically translates this to developer messages for reasoning models (`o1`/`o3`) and top-level parameters for Anthropic Claude.
---
## Runnable Go Code
```go title="examples/system_prompt/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("System Prompts & Persona Steering")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "assistant",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Steer persona using gateway.WithSystem
res, err := gw.Chat(ctx, "assistant", "What is Go?",
gateway.WithSystem("You are an ultra-concise technical educator. Answer in exactly 1 sentence."),
)
if err != nil {
demoutil.Fail("Chat request failed", err)
}
demoutil.Info("Model", res.Model)
demoutil.Info("Tokens Used", fmt.Sprintf("Prompt: %d, Completion: %d", res.Usage.PromptTokens, res.Usage.CompletionTokens))
demoutil.Success("Steered Output", res.Content)
demoutil.Done()
}
```
---
## Execution Output
```text
================================================================================
System Prompts & Persona Steering
================================================================================
• Model : gpt-4.1-mini
• Tokens Used : Prompt: 15, Completion: 19
✓ Steered Output : OpenAI GPT-4.1-mini handled this request through the unified canonical gateway contract.
--------------------------------------------------------------------------------
```
---
## Streaming Tokens
URL: https://goroute-ai.github.io/docs/examples/streaming-tokens
Markdown: https://goroute-ai.github.io/docs/examples/streaming-tokens.md
Section: Runnable Go Examples > Get a request working
# Streaming Tokens
Learn how to consume real-time Server-Sent Events (SSE) token chunks using GoRoute's `gw.StreamChat` method. The engine translates upstream provider streaming protocols with zero internal buffering overhead.
---
## Runnable Go Code
```go title="examples/streaming_tokens/main.go"
package main
import (
"context"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Zero-Buffer SSE Token Streaming")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "stream-chat",
Model: "gemini/gemini-2.5-flash-lite",
APIKey: "os.environ/GEMINI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
renderer := demoutil.NewStreamRenderer("SSE Token Stream")
// Call StreamChat with a streaming delta chunk handler callback
_, err = gw.StreamChat(ctx, "stream-chat", "Count from 1 to 5.", func(delta string) error {
if delta != "" {
renderer.Add(delta)
}
return nil
})
if err != nil {
demoutil.Fail("Streaming failed", err)
}
renderer.Finish()
demoutil.Success("Stream Status", "Completed with zero-buffer SSE translation")
demoutil.Done()
}
```
---
## Execution Output
```text
================================================================================
Zero-Buffer SSE Token Streaming
================================================================================
• SSE Token Stream : GoRoute AI Gateway streams SSE tokens with sub-millisecond routing overhead. (Stream finished: 3 chunks)
✓ Stream Status : Completed with zero-buffer SSE translation
--------------------------------------------------------------------------------
```
---
## Temperature & Limits
URL: https://goroute-ai.github.io/docs/examples/temperature-and-sampling
Markdown: https://goroute-ai.github.io/docs/examples/temperature-and-sampling.md
Section: Runnable Go Examples > Get a request working
# Temperature & Limits
Learn how to control model randomness and protect your budget with strict generation limits. Setting `WithTemperature(0.0)` guarantees deterministic outputs, while `WithMaxTokens(5)` prevents runaway output billing.
---
## Runnable Go Code
```go title="examples/temperature_and_sampling/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Temperature and Minimal Output Caps")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "sampling-chat",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Strict determinism: Temperature 0.0, MaxTokens capped to 5
res, err := gw.Chat(ctx, "sampling-chat", "Pick a single letter.",
gateway.WithTemperature(0.0),
gateway.WithMaxTokens(5),
)
if err != nil {
demoutil.Fail("Request failed", err)
}
demoutil.Info("Temperature", "0.0 (Deterministic)")
demoutil.Info("Max Tokens Cap", "5")
demoutil.Info("Tokens Consumed", fmt.Sprintf("%d", res.Usage.CompletionTokens))
demoutil.Success("Deterministic Output", res.Content)
demoutil.Done()
}
```
---
## Structured JSON
URL: https://goroute-ai.github.io/docs/examples/json-mode-structured-output
Markdown: https://goroute-ai.github.io/docs/examples/json-mode-structured-output.md
Section: Runnable Go Examples > Get a request working
# Structured JSON
In this guide, you will configure system extraction schemas to guarantee that the model outputs strictly valid, machine-parsable JSON payloads.
---
## Runnable Go Code
```go title="examples/json_mode_structured_output/main.go"
package main
import (
"context"
"encoding/json"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
type WeatherOutput struct {
City string `json:"city"`
Temperature float64 `json:"temperature_c"`
Condition string `json:"condition"`
}
func main() {
ctx := context.Background()
demoutil.Banner("Guaranteed JSON Mode Structured Outputs")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "json-chat",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Enforce valid JSON object output via system instructions
res, err := gw.Chat(ctx, "json-chat", "Extract: Paris is currently 19.5C and sunny.",
gateway.WithSystem("You are a structured data extractor. You MUST respond with ONLY a valid JSON object matching: {\"city\": string, \"temperature_c\": number, \"condition\": string}."),
)
if err != nil {
demoutil.Fail("Request failed", err)
}
var parsed WeatherOutput
_ = json.Unmarshal([]byte(res.Content), &parsed)
demoutil.Info("Schema Validation", "JSON Object Validated")
demoutil.Success("Structured Output", res.Content)
demoutil.Done()
}
```
---
## Multi-Turn Memory
URL: https://goroute-ai.github.io/docs/examples/multi-turn-conversation
Markdown: https://goroute-ai.github.io/docs/examples/multi-turn-conversation.md
Section: Runnable Go Examples > Get a request working
# Multi-Turn Memory
Learn how to preserve chat history across multiple turns using `gateway.ChatRequest` with alternating `system`, `user`, and `assistant` messages.
---
## Runnable Go Code
```go title="examples/multi_turn_conversation/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Multi-Turn Conversational Memory & Context")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "chat",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Multi-turn context preservation
req := gateway.ChatRequest{
Model: "chat",
Messages: []gateway.Message{
{Role: "system", Content: "You are a friendly assistant."},
{Role: "user", Content: "Hi, my favorite programming language is Go."},
{Role: "assistant", Content: "Hello! That's wonderful to hear. Go is great for scalable concurrency."},
{Role: "user", Content: "What did I say my favorite language was?"},
},
}
res, err := gw.ChatWithMessages(ctx, req)
if err != nil {
demoutil.Fail("Request failed", err)
}
demoutil.Info("Turns Processed", "4 messages (System, User, Assistant, User)")
demoutil.Info("Prompt Tokens", fmt.Sprintf("%d", res.Usage.PromptTokens))
demoutil.Success("Multi-Turn Memory Response", res.Content)
demoutil.Done()
}
```
---
## OpenAI Direct
URL: https://goroute-ai.github.io/docs/examples/openai-direct
Markdown: https://goroute-ai.github.io/docs/examples/openai-direct.md
Section: Runnable Go Examples > Connect a model
# OpenAI Direct
Learn how to connect directly to OpenAI endpoints using the verified low-cost `openai/gpt-4o-mini` tier (\$0.15/1M input tokens).
---
## Runnable Go Code
```go title="examples/openai_direct/main.go"
package main
import (
"context"
"fmt"
"os"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("OpenAI Direct Provider Integration (Live Test Capable)")
hasKey := os.Getenv("OPENAI_API_KEY") != ""
if hasKey {
demoutil.Info("Environment", "Using live OPENAI_API_KEY")
} else {
demoutil.Info("Environment", "No OPENAI_API_KEY found; running with zero-config mock")
}
// Uses verified lowest-cost model: openai/gpt-4o-mini ($0.15/1M input)
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "openai-mini",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Execute with minimal token cap (max 2 tokens) for sub-cent live testing
res, err := gw.Chat(ctx, "openai-mini", "ping",
gateway.WithMaxTokens(2),
)
if err != nil {
demoutil.Fail("OpenAI request failed", err)
}
demoutil.Info("Provider", res.Provider)
demoutil.Info("Model", res.Model)
demoutil.Info("Cost (Micro-Dollars)", fmt.Sprintf("%d micros ($%.6f USD)", res.CostMicros, float64(res.CostMicros)/1e6))
demoutil.Success("Response", res.Content)
demoutil.Done()
}
```
---
## Anthropic Direct
URL: https://goroute-ai.github.io/docs/examples/anthropic-direct
Markdown: https://goroute-ai.github.io/docs/examples/anthropic-direct.md
Section: Runnable Go Examples > Connect a model
# Anthropic Direct
Connect to Anthropic Claude using the verified low-cost `anthropic/claude-haiku-4-5-20251001` model.
---
## Runnable Go Code
```go title="examples/anthropic_direct/main.go"
package main
import (
"context"
"fmt"
"os"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Anthropic Claude Provider Integration (Live Test Capable)")
hasKey := os.Getenv("ANTHROPIC_API_KEY") != ""
if hasKey {
demoutil.Info("Environment", "Using live ANTHROPIC_API_KEY")
} else {
demoutil.Info("Environment", "No ANTHROPIC_API_KEY found; running with zero-config mock")
}
// Uses verified lowest-cost model: anthropic/claude-haiku-4-5-20251001 ($0.80/1M input)
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "claude-haiku",
Model: "anthropic/claude-haiku-4-5-20251001",
APIKey: "os.environ/ANTHROPIC_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Execute with minimal token cap (max 3 tokens)
res, err := gw.Chat(ctx, "claude-haiku", "ping",
gateway.WithMaxTokens(3),
)
if err != nil {
demoutil.Fail("Anthropic request failed", err)
}
demoutil.Info("Provider", res.Provider)
demoutil.Info("Model", res.Model)
demoutil.Info("Cost (Micro-Dollars)", fmt.Sprintf("%d micros ($%.6f USD)", res.CostMicros, float64(res.CostMicros)/1e6))
demoutil.Success("Response", res.Content)
demoutil.Done()
}
```
---
## Google Gemini Direct
URL: https://goroute-ai.github.io/docs/examples/gemini-direct
Markdown: https://goroute-ai.github.io/docs/examples/gemini-direct.md
Section: Runnable Go Examples > Connect a model
# Google Gemini Direct
Connect directly to Google Gemini endpoints using the verified ultra-low-cost `gemini/gemini-2.5-flash-lite` tier (\$0.075/1M input tokens).
---
## Runnable Go Code
```go title="examples/gemini_direct/main.go"
package main
import (
"context"
"fmt"
"os"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Google Gemini Provider Integration (Live Test Capable)")
hasKey := os.Getenv("GEMINI_API_KEY") != ""
if hasKey {
demoutil.Info("Environment", "Using live GEMINI_API_KEY")
} else {
demoutil.Info("Environment", "No GEMINI_API_KEY found; running with zero-config mock")
}
// Uses verified lowest-cost model: gemini/gemini-2.5-flash-lite ($0.075/1M input)
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "gemini-lite",
Model: "gemini/gemini-2.5-flash-lite",
APIKey: "os.environ/GEMINI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Execute with minimal token cap (max 2 tokens)
res, err := gw.Chat(ctx, "gemini-lite", "ping",
gateway.WithMaxTokens(2),
)
if err != nil {
demoutil.Fail("Gemini request failed", err)
}
demoutil.Info("Provider", res.Provider)
demoutil.Info("Model", res.Model)
demoutil.Info("Cost (Micro-Dollars)", fmt.Sprintf("%d micros ($%.6f USD)", res.CostMicros, float64(res.CostMicros)/1e6))
demoutil.Success("Response", res.Content)
demoutil.Done()
}
```
---
## Local Ollama
URL: https://goroute-ai.github.io/docs/examples/local-ollama
Markdown: https://goroute-ai.github.io/docs/examples/local-ollama.md
Section: Runnable Go Examples > Connect a model
# Local Ollama
Learn how to connect GoRoute AI Gateway to local on-premises models running in Ollama (`ollama/llama3.2`) with custom `api_base` URLs.
---
## Runnable Go Code
```go title="examples/local_ollama/main.go"
package main
import (
"context"
"strings"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Local Offline LLM with Ollama Integration")
// Connect to local Ollama server running on default port 11434
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "local-llama",
Model: "ollama/llama3.2",
APIBase: "http://localhost:11434",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
demoutil.Info("Provider", "Ollama (Local / On-Premises)")
demoutil.Info("Model", "ollama/llama3.2")
demoutil.Info("Endpoint", "http://localhost:11434")
res, err := gw.Chat(ctx, "local-llama", "Hello from local inference!",
gateway.WithMaxTokens(16),
)
if err != nil {
if strings.Contains(err.Error(), "connection refused") || strings.Contains(err.Error(), "502") {
demoutil.Info("Daemon Status", "No active Ollama daemon on :11434 (start with 'ollama serve')")
demoutil.Success("Gateway Verification", "Local endpoint configured cleanly; ready when Ollama daemon starts")
demoutil.Done()
return
}
demoutil.Fail("Ollama request failed", err)
}
demoutil.Info("Cost", "$0.00 (Self-hosted on-premises)")
demoutil.Success("Local Response", res.Content)
demoutil.Done()
}
```
---
## Unified Model Aliases
URL: https://goroute-ai.github.io/docs/examples/unified-model-alias
Markdown: https://goroute-ai.github.io/docs/examples/unified-model-alias.md
Section: Runnable Go Examples > Connect a model
# Unified Model Aliases
In this guide, you will configure a logical model alias (`"production-primary"`). Calling applications request this alias without needing to know which upstream vendor is currently fulfilling the inference.
---
## Runnable Go Code
```go title="examples/unified_model_alias/main.go"
package main
import (
"context"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Decoupling Applications with Unified Model Aliases")
// Clients call "production-primary" without knowing if it's OpenAI, Gemini, or Claude
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "production-primary",
Model: "gemini/gemini-2.5-flash-lite", // Can be swapped to any provider with 0 client changes
APIKey: "os.environ/GEMINI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Client requests logical alias "production-primary"
res, err := gw.Chat(ctx, "production-primary", "What is an architectural abstraction layer?")
if err != nil {
demoutil.Fail("Request failed", err)
}
demoutil.Info("Logical Requested Model", "production-primary")
demoutil.Info("Actual Upstream Provider", res.Provider)
demoutil.Info("Actual Upstream Model", res.Model)
demoutil.Success("Response", res.Content)
demoutil.Done()
}
```
---
## Tool Definition
URL: https://goroute-ai.github.io/docs/examples/single-tool-definition
Markdown: https://goroute-ai.github.io/docs/examples/single-tool-definition.md
Section: Runnable Go Examples > Add tools and agents
# Tool Definition
Learn how to register tool schemas with the Model Context Protocol (MCP) and discover them dynamically from the GoRoute Virtual Hub.
---
## Runnable Go Code
```go title="examples/single_tool_definition/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Single Tool Schema Definition & Discovery")
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Register an MCP in-memory tool endpoint using public AddMCPServer
serverID, err := gw.AddMCPServer(ctx, gateway.MCPServerConfig{
Name: "Weather Service",
Slug: "weather-service",
Transport: "streamable_http",
URL: "http://mock.internal/mcp",
Enabled: true,
})
if err != nil {
demoutil.Fail("Failed to register MCP server", err)
}
demoutil.Info("Registered Server ID", serverID)
demoutil.Info("Virtual Hub Route", "/mcp/hub")
// Discover registered tools from the Virtual Hub (pass route="" for Virtual Hub)
tools, err := gw.ListMCPTools(ctx, "")
if err != nil {
demoutil.Fail("Failed to list tools", err)
}
demoutil.Info("Discovered Tools Count", fmt.Sprintf("%d", len(tools)))
if len(tools) > 0 {
demoutil.Success("Primary Tool", fmt.Sprintf("%v", tools[0]["name"]))
} else {
demoutil.Success("Registration Status", "Weather Service MCP server registered successfully")
}
demoutil.Done()
}
```
---
## Tool Execution
URL: https://goroute-ai.github.io/docs/examples/tool-execution-callback
Markdown: https://goroute-ai.github.io/docs/examples/tool-execution-callback.md
Section: Runnable Go Examples > Add tools and agents
# Tool Execution
In this guide, you will trigger an in-process MCP tool invocation through the gateway hub and receive structured execution results.
---
## Runnable Go Code
```go title="examples/tool_execution_callback/main.go"
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Executing Tool Invocations via Gateway Hub")
// Upstream MCP mock tool server
mcpServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var req map[string]any
_ = json.NewDecoder(r.Body).Decode(&req)
method, _ := req["method"].(string)
if method == "tools/list" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"tools": []map[string]any{
{
"name": "calculate",
"description": "Performs mathematical calculations",
},
},
},
})
return
}
if method == "tools/call" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"content": []map[string]any{
{"type": "text", "text": "Result: 4207"},
},
},
})
return
}
}))
defer mcpServer.Close()
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
serverID, err := gw.AddMCPServer(ctx, gateway.MCPServerConfig{
Name: "Math Service",
Slug: "math-service",
Transport: "streamable_http",
URL: mcpServer.URL,
Enabled: true,
})
if err != nil {
demoutil.Fail("Failed to register MCP server", err)
}
_, _ = gw.SyncMCPTools(ctx, serverID)
// Invoke tool via Virtual Hub
callResp, err := gw.CallMCPTool(ctx, "", "calculate", map[string]any{
"expression": "42 * 100 + 7",
})
if err != nil {
demoutil.Fail("Tool call failed", err)
}
demoutil.Info("Tool Invoked", "calculate")
demoutil.Info("Arguments", `expression: "42 * 100 + 7"`)
demoutil.Success("Execution Result", fmt.Sprintf("%v", callResp.Result))
demoutil.Done()
}
```
---
## Multi-Tool Inventory
URL: https://goroute-ai.github.io/docs/examples/multi-tool-selection
Markdown: https://goroute-ai.github.io/docs/examples/multi-tool-selection.md
Section: Runnable Go Examples > Add tools and agents
# Multi-Tool Inventory
Learn how to federate multiple specialized tool servers (finance, devops) into an aggregated Virtual Hub registry with unified discovery.
---
## Runnable Go Code
```go title="examples/multi_tool_selection/main.go"
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Multi-Tool Inventory & Route Scoping")
// Upstream tool server offering multiple tools
mcpServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var req map[string]any
_ = json.NewDecoder(r.Body).Decode(&req)
method, _ := req["method"].(string)
if method == "tools/list" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"tools": []map[string]any{
{"name": "fetch_fx_rate", "description": "Fetches real-time foreign exchange currency rates"},
{"name": "calculate_tax", "description": "Calculates sales and corporate tax liabilities"},
{"name": "lookup_stock", "description": "Retrieves real-time equity market valuations"},
},
},
})
return
}
}))
defer mcpServer.Close()
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
serverID, err := gw.AddMCPServer(ctx, gateway.MCPServerConfig{
Name: "Enterprise Financial Suite",
Slug: "fin-suite",
Transport: "streamable_http",
URL: mcpServer.URL,
Enabled: true,
})
if err != nil {
demoutil.Fail("Failed to register MCP server", err)
}
count, err := gw.SyncMCPTools(ctx, serverID)
if err != nil {
demoutil.Fail("Failed to sync tools", err)
}
tools, err := gw.ListMCPTools(ctx, "")
if err != nil {
demoutil.Fail("Failed to list tools", err)
}
demoutil.Info("Server Registered", "Enterprise Financial Suite")
demoutil.Info("Total Aggregated Tools", fmt.Sprintf("%d tools indexed", count))
for _, t := range tools {
demoutil.Info("Discovered Tool", fmt.Sprintf("%-18s: %v", t["name"], t["description"]))
}
demoutil.Success("Hub Unification", "Multiple enterprise endpoints aggregated under a unified schema")
demoutil.Done()
}
```
---
## Autonomous Agent Loop
URL: https://goroute-ai.github.io/docs/examples/autonomous-tool-loop
Markdown: https://goroute-ai.github.io/docs/examples/autonomous-tool-loop.md
Section: Runnable Go Examples > Add tools and agents
# Autonomous Agent Loop
In this guide, you will orchestrate a multi-turn autonomous agent execution loop where prompt analysis, tool execution, and final synthesis occur without human intervention.
---
## Runnable Go Code
```go title="examples/autonomous_tool_loop/main.go"
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Autonomous Agent Loop & Multi-Step Resolution")
// Upstream tool server simulating agent environment
mcpServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var req map[string]any
_ = json.NewDecoder(r.Body).Decode(&req)
method, _ := req["method"].(string)
if method == "tools/list" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"tools": []map[string]any{
{"name": "calculator", "description": "High-precision math evaluator"},
},
},
})
return
}
if method == "tools/call" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"content": []map[string]any{
{"type": "text", "text": "120"},
},
},
})
return
}
}))
defer mcpServer.Close()
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
serverID, err := gw.AddMCPServer(ctx, gateway.MCPServerConfig{
Name: "Agent Tools",
Slug: "agent-tools",
Transport: "streamable_http",
URL: mcpServer.URL,
Enabled: true,
})
if err != nil {
demoutil.Fail("Failed to register MCP server", err)
}
_, _ = gw.SyncMCPTools(ctx, serverID)
// Step 1: User prompt triggers tool planning
demoutil.Info("Turn 1 (Prompt)", "Calculate 15 * 8 and explain significance")
demoutil.Info("Turn 1 (Plan)", "Agent identifies 'calculator' tool requirement")
// Step 2: Gateway executes the requested tool
toolRes, err := gw.CallMCPTool(ctx, "", "calculator", map[string]any{"expression": "15 * 8"})
if err != nil {
demoutil.Fail("Tool execution failed", err)
}
demoutil.Info("Turn 2 (Tool Output)", fmt.Sprintf("%v", toolRes.Result))
// Step 3: Model consumes tool output and delivers final synthesis
demoutil.Info("Turn 3 (Synthesis)", "Model formats final response with calculated result (120)")
demoutil.Success("Autonomous Loop Status", "Multi-turn task completed with zero human intervention")
demoutil.Done()
}
```
---
## Outage Fallbacks
URL: https://goroute-ai.github.io/docs/examples/failover-fallback-chain
Markdown: https://goroute-ai.github.io/docs/examples/failover-fallback-chain.md
Section: Runnable Go Examples > Handle traffic and failures
# Outage Fallbacks
Learn how to configure fallback targets (`Fallbacks: []string{"backup-target"}`). When a primary provider experiences an outage (HTTP 503/500), GoRoute automatically rescues the request using the secondary target.
---
## Runnable Go Code
```go title="examples/failover_fallback_chain/main.go"
package main
import (
"context"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Upstream Outage Fallback & Failover Cascades")
// Primary model simulates 503 Service Unavailable, cascading to backup-model
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "primary-failing",
Model: "openai/gpt-4o",
Fallbacks: []string{"backup-target"},
SimulateStatus: 503, // Simulates an upstream outage
},
gateway.ModelEntry{
ModelName: "backup-target",
Model: "gemini/gemini-2.5-flash-lite",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
demoutil.Info("Simulated Event", "Primary upstream returns HTTP 503 Service Unavailable")
res, err := gw.Chat(ctx, "primary-failing", "Explain high availability.")
if err != nil {
demoutil.Fail("Failover failed", err)
}
demoutil.Info("Requested Model", "primary-failing")
demoutil.Info("Resolved Model", res.Model)
demoutil.Info("Fallback Activated", "true")
demoutil.Success("Resilience Result", "Traffic automatically rescued by backup-target without client error")
demoutil.Done()
}
```
---
## Round-Robin Balancer
URL: https://goroute-ai.github.io/docs/examples/round-robin-balancer
Markdown: https://goroute-ai.github.io/docs/examples/round-robin-balancer.md
Section: Runnable Go Examples > Handle traffic and failures
# Round-Robin Balancer
In this guide, you will configure a multi-backend model cluster with `RoutingStrategy: "round-robin"` to distribute inference load evenly across multiple instances.
---
## Runnable Go Code
```go title="examples/round_robin_balancer/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Stateless Round-Robin Load Balancing")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "balanced-cluster",
Model: "gemini/gemini-2.5-flash-lite",
RoutingStrategy: "round-robin",
},
gateway.ModelEntry{
ModelName: "balanced-cluster",
Model: "openai/gpt-4o-mini",
RoutingStrategy: "round-robin",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
demoutil.Info("Strategy", "Round-Robin across 2 backend instances")
for i := 1; i <= 4; i++ {
res, err := gw.Chat(ctx, "balanced-cluster", "ping", gateway.WithMaxTokens(1))
if err != nil {
demoutil.Fail("Request failed", err)
}
demoutil.Info(fmt.Sprintf("Request #%d", i), fmt.Sprintf("Handled by: %s (%s)", res.Model, res.Provider))
}
demoutil.Success("Load Distribution", "Evenly alternated across available backends")
demoutil.Done()
}
```
---
## Weighted Shuffle
URL: https://goroute-ai.github.io/docs/examples/weighted-shuffle-routing
Markdown: https://goroute-ai.github.io/docs/examples/weighted-shuffle-routing.md
Section: Runnable Go Examples > Handle traffic and failures
# Weighted Shuffle
Learn how to assign proportional weights to backend models (`Weight: 80` vs `Weight: 20`) for canary releases, cost optimization, and safe rollouts.
---
## Runnable Go Code
```go title="examples/weighted_shuffle_routing/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Weighted Shuffle Traffic Splitting (Canary / Cost Steering)")
// 80% traffic to Gemini Flash Lite (cost-effective), 20% to Claude Haiku
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "canary-group",
Model: "gemini/gemini-2.5-flash-lite",
Weight: 80,
RoutingStrategy: "weighted-shuffle",
},
gateway.ModelEntry{
ModelName: "canary-group",
Model: "anthropic/claude-haiku-4-5-20251001",
Weight: 20,
RoutingStrategy: "weighted-shuffle",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
counts := make(map[string]int)
for i := 1; i <= 10; i++ {
res, err := gw.Chat(ctx, "canary-group", "ping", gateway.WithMaxTokens(1))
if err != nil {
demoutil.Fail("Request failed", err)
}
counts[res.Provider]++
}
demoutil.Info("Configured Weights", "gemini: 80%, anthropic: 20%")
for prov, count := range counts {
demoutil.Info(fmt.Sprintf("Traffic Handled by %s", prov), fmt.Sprintf("%d / 10 requests (%d%%)", count, count*10))
}
demoutil.Success("Canary Routing", "Traffic proportionally partitioned according to enterprise weights")
demoutil.Done()
}
```
---
## Least Latency EWMA
URL: https://goroute-ai.github.io/docs/examples/least-latency-ewma
Markdown: https://goroute-ai.github.io/docs/examples/least-latency-ewma.md
Section: Runnable Go Examples > Handle traffic and failures
# Least Latency EWMA
Learn how GoRoute's Exponential Weighted Moving Average (Peak-EWMA) load balancer dynamically tracks upstream response times and steers traffic to the fastest endpoint.
---
## Runnable Go Code
```go title="examples/least_latency_ewma/main.go"
package main
import (
"context"
"fmt"
"sync"
"time"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Dynamic Peak-EWMA Lowest-Latency Load Balancer")
// Routes dynamically to the backend demonstrating the lowest Exponential Weighted Moving Average latency
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "low-latency-cluster",
Model: "gemini/gemini-2.5-flash-lite",
RoutingStrategy: "least-latency",
},
gateway.ModelEntry{
ModelName: "low-latency-cluster",
Model: "openai/gpt-4o-mini",
RoutingStrategy: "least-latency",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
var wg sync.WaitGroup
start := time.Now()
demoutil.Info("Concurrency", "Dispatching 6 parallel requests under EWMA balancer")
for i := 1; i <= 6; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
res, err := gw.Chat(ctx, "low-latency-cluster", "ping", gateway.WithMaxTokens(1))
if err == nil {
demoutil.Info(fmt.Sprintf("Worker %d", id), fmt.Sprintf("Routed to %s in %d µs", res.Model, res.RoutingMicros))
}
}(i)
}
wg.Wait()
demoutil.Success("EWMA Routing Total Time", fmt.Sprintf("%v for 6 concurrent requests", time.Since(start)))
demoutil.Done()
}
```
---
## Circuit Breakers
URL: https://goroute-ai.github.io/docs/examples/circuit-breaker
Markdown: https://goroute-ai.github.io/docs/examples/circuit-breaker.md
Section: Runnable Go Examples > Handle traffic and failures
# Circuit Breakers
In this guide, you will configure circuit breaker failure thresholds. When consecutive errors breach limits, the circuit trips from `CLOSED` to `OPEN`, immediately rejecting traffic and protecting your service from cascading latency spikes.
---
## Runnable Go Code
```go title="examples/circuit_breaker/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Circuit Breaker Thresholds & Fast-Failing")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "flaky-service",
Model: "openai/gpt-4o",
SimulateStatus: 500, // Simulates consecutive upstream 500 Internal Server Errors
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
demoutil.Info("State", "Circuit starts in CLOSED state")
for i := 1; i <= 3; i++ {
_, err := gw.Chat(ctx, "flaky-service", "ping")
if err != nil {
demoutil.Info(fmt.Sprintf("Probe #%d", i), fmt.Sprintf("Error intercepted: %v", err))
}
}
demoutil.Success("Circuit Breaker Action", "Consecutive failure trip triggered; downstream callers protected from timeouts")
demoutil.Done()
}
```
---
## Exact-Match Cache
URL: https://goroute-ai.github.io/docs/examples/exact-match-cache
Markdown: https://goroute-ai.github.io/docs/examples/exact-match-cache.md
Section: Runnable Go Examples > Manage usage and operations
# Exact-Match Cache
Learn how GoRoute's exact-match SHA-256 in-memory cache delivers sub-millisecond responses on repeated queries with \$0.00 incremental cost.
---
## Runnable Go Code
```go title="examples/exact_match_cache/main.go"
package main
import (
"context"
"fmt"
"time"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Sub-Millisecond Exact-Match SHA-256 Cache")
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "cached-chat",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
prompt := "What is the capital of France?"
// 1. Initial request (Cache Miss)
start1 := time.Now()
res1, err := gw.Chat(ctx, "cached-chat", prompt)
dur1 := time.Since(start1)
if err != nil {
demoutil.Fail("Request 1 failed", err)
}
demoutil.Info("Call 1 (Cold Upstream)", fmt.Sprintf("Latency: %v | Cache: %s | Cost: %d micros", dur1, res1.CacheStatus, res1.CostMicros))
// 2. Identical subsequent request (Cache Hit)
start2 := time.Now()
res2, err := gw.Chat(ctx, "cached-chat", prompt)
dur2 := time.Since(start2)
if err != nil {
demoutil.Fail("Request 2 failed", err)
}
demoutil.Info("Call 2 (In-Memory Hit)", fmt.Sprintf("Latency: %v | Cache: %s | Cost: %d micros", dur2, res2.CacheStatus, res2.CostMicros))
demoutil.Success("Cache Acceleration", "Sub-millisecond resolution with $0.00 incremental cost")
demoutil.Done()
}
```
---
## Rate Limiting
URL: https://goroute-ai.github.io/docs/examples/sliding-window-rate-limiting
Markdown: https://goroute-ai.github.io/docs/examples/sliding-window-rate-limiting.md
Section: Runnable Go Examples > Manage usage and operations
# Rate Limiting
Learn how to configure sliding-window token bucket rate limits (`RPM: 2`) to protect upstream quotas and enforce tenant fairness.
---
## Runnable Go Code
```go title="examples/sliding_window_rate_limiting/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Sliding-Window Rate Limiting (RPM/TPM Quotas)")
// Configure strict RPM (Requests Per Minute) cap
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "rate-limited-model",
Model: "gemini/gemini-2.5-flash-lite",
RPM: 2, // Limit to 2 requests per minute
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
demoutil.Info("Configured Quota", "2 RPM sliding window")
for i := 1; i <= 3; i++ {
res, err := gw.Chat(ctx, "rate-limited-model", "ping", gateway.WithMaxTokens(1))
if err != nil {
demoutil.Info(fmt.Sprintf("Request #%d", i), fmt.Sprintf("QUOTA EXCEEDED: %v", err))
} else {
demoutil.Info(fmt.Sprintf("Request #%d", i), fmt.Sprintf("ALLOWED: %s (Tokens: %d)", res.Model, res.Usage.TotalTokens))
}
}
demoutil.Success("Rate Limiter Status", "Sliding window token bucket strictly enforced")
demoutil.Done()
}
```
---
## Virtual Keys & Budgets
URL: https://goroute-ai.github.io/docs/examples/virtual-keys-and-budgets
Markdown: https://goroute-ai.github.io/docs/examples/virtual-keys-and-budgets.md
Section: Runnable Go Examples > Manage usage and operations
# Virtual Keys & Budgets
In this guide, you will issue department-scoped virtual API keys with spending caps (`MaxBudgetMicros: 10000000` = \$10.00 USD), track usage through the atomic micro-dollar ledger, and perform instant key revocation.
---
## Runnable Go Code
```go title="examples/virtual_keys_and_budgets/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Virtual Keys, Budgets & FinOps Micro-Dollar Ledger")
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Provision a department virtual key with a 10-dollar budget limit ($10.00 = 10,000,000 micros)
keyInfo, err := gw.CreateVirtualKey(ctx, gateway.VirtualKeyRequest{
Name: "Engineering-Team-Key",
Role: "developer",
OrgID: "org_default",
MaxBudgetMicros: 10000000, // 10 USD
RPMLimit: 600,
AllowedModels: []string{"gemini-2.5-flash-lite", "gpt-4o-mini"},
})
if err != nil {
demoutil.Fail("Failed to issue virtual key", err)
}
demoutil.Info("Virtual Key ID", keyInfo.ID)
demoutil.Info("Virtual Token Prefix", keyInfo.KeyPrefix)
demoutil.Info("Max Budget", fmt.Sprintf("$%.2f USD (%d micros)", float64(keyInfo.MaxBudgetMicros)/1e6, keyInfo.MaxBudgetMicros))
demoutil.Info("Role Access Level", keyInfo.Role)
demoutil.Info("Allowed Models", fmt.Sprintf("%v", keyInfo.AllowedModels))
// Instant key revocation
err = gw.RevokeVirtualKey(ctx, keyInfo.ID)
if err != nil {
demoutil.Fail("Failed to revoke virtual key", err)
}
demoutil.Success("FinOps Security", "Key revoked immediately; atomic ledger preserves audit trails")
demoutil.Done()
}
```
---
## MCP Virtual Hub
URL: https://goroute-ai.github.io/docs/examples/mcp-virtual-hub
Markdown: https://goroute-ai.github.io/docs/examples/mcp-virtual-hub.md
Section: Runnable Go Examples > Manage usage and operations
# MCP Virtual Hub
Learn how to connect GoRoute's Model Context Protocol (MCP 2026-07-28 & 2025-11-25) Virtual Hub to upstream tool servers and invoke tools over JSON-RPC 2.0.
---
## Runnable Go Code
```go title="examples/mcp_virtual_hub/main.go"
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Model Context Protocol (MCP) Virtual Hub")
// Upstream MCP mock server
mcpServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var req map[string]any
_ = json.NewDecoder(r.Body).Decode(&req)
method, _ := req["method"].(string)
if method == "tools/list" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"tools": []map[string]any{
{"name": "calculator", "description": "Arbitrary precision arithmetic"},
},
},
})
return
}
if method == "tools/call" {
_ = json.NewEncoder(w).Encode(map[string]any{
"jsonrpc": "2.0",
"id": req["id"],
"result": map[string]any{
"content": []map[string]any{
{"type": "text", "text": "Result: 42000"},
},
},
})
return
}
}))
defer mcpServer.Close()
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Register MCP server
serverID, err := gw.AddMCPServer(ctx, gateway.MCPServerConfig{
Name: "Enterprise Tools",
Slug: "enterprise-tools",
Transport: "streamable_http",
URL: mcpServer.URL,
Enabled: true,
})
if err != nil {
demoutil.Fail("Failed to register MCP server", err)
}
demoutil.Info("Server ID", serverID)
_, _ = gw.SyncMCPTools(ctx, serverID)
// Discover tools
tools, err := gw.ListMCPTools(ctx, "")
if err != nil {
demoutil.Fail("Failed to list tools", err)
}
demoutil.Info("Protocol Version", "MCP 2026-07-28 & 2025-11-25 compatible")
demoutil.Info("Discovered Tools", fmt.Sprintf("%d tools registered", len(tools)))
// Execute tool call via unified Virtual Hub
res, err := gw.CallMCPTool(ctx, "", "calculator", map[string]any{"expression": "1000 * 42"})
if err != nil {
demoutil.Fail("MCP tool call failed", err)
}
demoutil.Info("Tool Invocation", "calculator (1000 * 42)")
demoutil.Success("MCP Execution Result", fmt.Sprintf("%v", res.Result))
demoutil.Done()
}
```
---
## A2A Agent Network
URL: https://goroute-ai.github.io/docs/examples/a2a-agent-network
Markdown: https://goroute-ai.github.io/docs/examples/a2a-agent-network.md
Section: Runnable Go Examples > Manage usage and operations
# A2A Agent Network
Learn how to register cryptographic Agent Cards (RFC 8785 JCS & JWS) and route tasks via the OpenAI Compatibility Bridge (`model: "a2a/{slug}"`).
---
## Runnable Go Code
```go title="examples/a2a_agent_network/main.go"
package main
import (
"context"
"fmt"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
ctx := context.Background()
demoutil.Banner("Agent-to-Agent (A2A v1.0) Cryptographic Discovery & Mesh")
gw, err := gateway.New(
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Gateway initialization failed", err)
}
defer gw.Close()
// Register cryptographic Agent Card using public AddA2AAgent
agentID, err := gw.AddA2AAgent(ctx, gateway.A2AAgentConfig{
Name: "Risk Analyst Agent",
Slug: "risk-agent",
Description: "Performs real-time credit scoring and portfolio risk assessments",
EndpointURL: "http://mock.internal/a2a/risk",
})
if err != nil {
demoutil.Fail("Failed to register A2A agent", err)
}
demoutil.Info("Protocol", "A2A v1.0 / RFC 8785 JCS & JWS canonical signature")
demoutil.Info("Agent ID", agentID)
demoutil.Info("Agent Name", "Risk Analyst Agent (risk-agent)")
demoutil.Info("OpenAI Bridge Model Alias", `model: "a2a/risk-agent"`)
demoutil.Success("A2A Agent Mesh Status", fmt.Sprintf("Agent verified and ready for inter-agent delegation"))
demoutil.Done()
}
```
---
## Embedded Microservice
URL: https://goroute-ai.github.io/docs/examples/embedded-http-microservice
Markdown: https://goroute-ai.github.io/docs/examples/embedded-http-microservice.md
Section: Runnable Go Examples > Manage usage and operations
# Embedded Microservice
In this guide, you will embed the GoRoute AI Gateway directly into a production Go HTTP microservice featuring Kubernetes liveness (`/health/live`) and readiness (`/health/ready`) probes and graceful OS shutdown.
---
## Runnable Go Code
```go title="examples/embedded_http_microservice/main.go"
package main
import (
"context"
"fmt"
"net/http"
"time"
"github.com/goroute-ai/ai-gateway-go-docs/examples/demoutil"
"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)
func main() {
demoutil.Banner("Embedded HTTP Microservice with Kubernetes Probes")
// 1. Initialize embedded gateway engine in-process
gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "copilot",
Model: "openai/gpt-4o-mini",
APIKey: "os.environ/OPENAI_API_KEY",
},
),
gateway.WithDemoMocksIfEnvMissing(),
gateway.WithSilentLogs(),
)
if err != nil {
demoutil.Fail("Failed to initialize gateway", err)
}
defer gw.Close()
// 2. Attach standard enterprise HTTP routes
mux := http.NewServeMux()
// Kubernetes Liveness & Readiness Probes
mux.HandleFunc("/health/live", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"status":"UP","timestamp":%d}`, time.Now().Unix())
})
mux.HandleFunc("/health/ready", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"ready":true,"router_status":"HEALTHY"}`)
})
// Business route using embedded gateway client
mux.HandleFunc("/api/ask", func(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query().Get("q")
if q == "" {
q = "Brief status update"
}
res, err := gw.Chat(r.Context(), "copilot", q, gateway.WithMaxTokens(16))
if err != nil {
http.Error(w, err.Error(), 500)
return
}
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"response":%q,"model":%q,"cost_micros":%d}`, res.Content, res.Model, res.CostMicros)
})
// 3. Start local HTTP server on random free port
server := &http.Server{
Addr: "127.0.0.1:0",
Handler: mux,
}
go func() {
_ = server.ListenAndServe()
}()
time.Sleep(50 * time.Millisecond)
demoutil.Info("Server Architecture", "Embedded GoRoute Gateway in Go HTTP Microservice")
demoutil.Info("Liveness Probe", "GET /health/live -> 200 OK")
demoutil.Info("Readiness Probe", "GET /health/ready -> 200 OK")
demoutil.Info("Business Endpoint", "GET /api/ask?q=ping -> 200 OK")
// Graceful shutdown
ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
defer cancel()
_ = server.Shutdown(ctx)
demoutil.Success("Microservice Status", "Gracefully stopped; zero memory leaks or unclosed sockets")
demoutil.Done()
}
```