Observability, Metrics & Health Probes
Observability helps you answer practical questions: did a request reach a provider, how long did it take, and is the service healthy? GoRoute can expose metrics, pass distributed trace context, and report health information for operators and monitoring tools.
The detailed setup includes:
- Prometheus Metrics Engine (
GET /metrics): In-process Vitter Algorithm R reservoir sampling (1,024samples) of latencies, Time-To-First-Token (TTFT), routing overheads, cache-hit ratios, and provider spend. - OTLP/HTTP & W3C Distributed Tracing: Exports
gen_ai.clientspans (gen_ai.operation.name,gen_ai.system,gen_ai.request.model,gen_ai.usage.*) viatracing.otlp_endpointthrough the outbound SSRF policy, alongsidetraceparentandX-Trace-Idpropagation. - Multi-Tier Health Probes & Policy Simulation: Kubernetes liveness (
/healthz), readiness (/readyz), active upstream deployment probing (/admin/v1/health/*), and the stateless Guardrails Policy Simulator (POST /admin/v1/organizations/{orgID}/guardrails/simulate). - Embedded Operator Console & Secret-Scrubbed Logs: Single-binary React UI served at
/ui/and structuredsloglogs scrubbed viainternal/logger/scrub.go(ScrubSecrets).
See also: OpenTelemetry & Prometheus Integration, Embedded Operator Web UI, and Troubleshooting.
1. Prometheus Metrics Scraper (GET /metrics)
The gateway exposes a protected Prometheus scraper endpoint at GET /metrics formatted according to Prometheus exposition specification 0.0.4. Requests require the Master Admin Key via Authorization: Bearer <MASTER_ADMIN_KEY>.
Metrics are maintained in-memory using Vitter Algorithm R uniform reservoir sampling (1,024 samples) and omit per-deployment labels to keep metric cardinality bounded.
Exported Metrics Catalog
| Metric Name | Type | Labels | Description |
|---|---|---|---|
gateway_requests_total | Counter | None | Total inference requests processed across all endpoints. |
gateway_errors_total | Counter | None | Total failed inference requests (HTTP 4xx/5xx). |
gateway_latency_ms | Gauge | quantile="0.5", "0.95", "0.99" | End-to-end request duration percentiles in milliseconds. |
gateway_ttft_ms | Gauge | quantile="0.95" | P95 Time-To-First-Token for streaming responses in milliseconds. |
gateway_routing_overhead_micros | Gauge | quantile="0.95" | P95 Stage 1 (heuristic) + Stage 2 (load balancer) routing decision time in microseconds. |
gateway_fallbacks_total | Counter | None | Total failover executions to secondary or tertiary model groups. |
gateway_retries_total | Counter | None | Total intra-group exponential retry attempts. |
gateway_cache_hits_total | Counter | None | Total exact-match responses served directly from Redis/L1 cache. |
gateway_ratelimit_rejections_total | Counter | None | Total requests rejected by RPM/TPM rate limits or financial spend ceilings. |
gateway_guardrail_blocks_total | Counter | None | Total requests intercepted and blocked by security guardrail rules. |
gateway_guardrail_evaluations_total | Counter | guardrail, hook, decision | Total guardrail evaluations segmented by guardrail name, execution hook, and decision (allow, block, flag, modify, error). |
gateway_guardrail_errors_total | Counter | guardrail, hook | Total guardrail evaluation errors segmented by guardrail name and hook. |
gateway_guardrail_duration_ms | Histogram | guardrail, hook | Guardrail evaluation latency histogram in milliseconds (le buckets and +Inf). |
gateway_provider_requests_total | Counter | provider | Request volume segmented by upstream provider (openai, anthropic, etc.). |
gateway_provider_errors_total | Counter | provider | Upstream failure count segmented by provider. |
gateway_provider_latency_p95_ms | Gauge | provider | P95 upstream round-trip latency segmented by provider. |
gateway_provider_spend_micros_total | Counter | provider | Cumulative spend in integer micro-dollars (10^-6 USD) per provider. |
Optional OTLP/HTTP Tracing & Structured Log Scrubbing
- OTLP Distributed Tracing: Configure
tracing.otlp_endpoint(and optionaltracing.headers) to export OpenTelemetry spans withgen_ai.*attributes through the outbound SSRF network policy. - Structured Log Scrubbing:
internal/logger/scrub.go(ScrubSecrets) masks API keys, bearer tokens, and credential patterns across structuredslogfields before logs are emitted.
Sample Scraper Output
# HELP gateway_requests_total Total inference requests processed by the AI Gateway.
# TYPE gateway_requests_total counter
gateway_requests_total 42180
# HELP gateway_errors_total Total failed requests.
# TYPE gateway_errors_total counter
gateway_errors_total 12
# HELP gateway_latency_ms Request latency percentiles in milliseconds.
# TYPE gateway_latency_ms gauge
gateway_latency_ms{quantile="0.5"} 342
gateway_latency_ms{quantile="0.95"} 890
gateway_latency_ms{quantile="0.99"} 1420
# HELP gateway_ttft_ms Time-To-First-Token in milliseconds.
# TYPE gateway_ttft_ms gauge
gateway_ttft_ms{quantile="0.95"} 184
# HELP gateway_routing_overhead_micros Stage 1 + Stage 2 routing decision overhead in microseconds.
# TYPE gateway_routing_overhead_micros gauge
gateway_routing_overhead_micros{quantile="0.95"} 42
# HELP gateway_cache_hits_total Total exact-match response cache hits.
# TYPE gateway_cache_hits_total counter
gateway_cache_hits_total 8520
# HELP gateway_provider_requests_total Total requests by provider.
# TYPE gateway_provider_requests_total counter
gateway_provider_requests_total{provider="anthropic"} 21040
gateway_provider_requests_total{provider="gemini"} 15300
gateway_provider_requests_total{provider="openai"} 5840
Scraping via Prometheus (prometheus.yml)
Add the following scrape configuration to your Prometheus server:
scrape_configs:
- job_name: "goroute-ai-gateway"
scrape_interval: 15s
static_configs:
- targets: ["gateway-service.default.svc.cluster.local:8080"]
metrics_path: "/metrics"
2. W3C Distributed Tracing & Header Propagation
To ensure transparent traceability across distributed microservice topologies, the gateway natively implements the W3C Trace Context specification (traceparent).
Trace Identifier Generation (internal/observability/tracing.go)
Every incoming HTTP request is evaluated by TraceMiddleware:
- Trace ID Ingestion: Checks for an existing W3C
traceparentheader (format:00-{trace_id}-{span_id}-{flags}). If absent, checksX-Trace-Id. If neither is provided, generates a fresh 128-bit random Trace ID (32 hex characters). - Span ID Generation: Generates a new 64-bit random Span ID (16 hex characters) representing the gateway's processing span.
- Outbound Header Injection: Automatically injects
traceparentandX-Trace-Idinto:- Client response headers.
- Outbound HTTP requests sent to upstream cloud providers (OpenAI, Anthropic, Bedrock, Vertex).
- Webhook push notifications (A2A transactional outbox).
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
X-Trace-Id: 4bf92f3577b34da6a3ce929d0e0e4736
X-Gateway-Request-Id: req_8f12a39b4c0e
Upstream Attribution Telemetry Headers
Every data plane response includes standard gateway telemetry headers:
| Header Name | Value Format | Description |
|---|---|---|
X-Trace-Id | 32-char hex | W3C 128-bit distributed trace correlation ID. |
X-Gateway-Request-Id | req_<uuid> | Unique per-request gateway tracking ID. |
X-Gateway-Provider | string | Resolved upstream provider (e.g. anthropic, gemini). |
X-Gateway-Model | string | Physical upstream model invoked (e.g. claude-3-5-sonnet-20241022). |
X-Gateway-Cost-Micros | integer | Micro-dollars (10^-6 USD) charged for the operation. |
X-Gateway-Routing-Time | duration | Microseconds elapsed making Stage 1 + 2 routing decisions. |
X-Gateway-Cache | HIT / MISS | Indicates whether the response was served from cache. |
3. Deployment Health Probes
The gateway provides three distinct health check interfaces designed for load balancers, orchestrators, and automated site reliability tooling.
Liveness Probe (GET /healthz)
A lightweight probe returning HTTP 200 as long as the Go HTTP process is running:
curl -i http://localhost:8080/healthz
Response:
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
X-Trace-Id: c8901f43a0e14cb789123456789abcde
OK
Readiness Probe (GET /readyz)
Verifies that all configured stateful storage dependencies are operational. When running in postgres mode, the probe executes an active database ping and verifies Redis connectivity.
curl -i http://localhost:8080/readyz
- If both PostgreSQL and Redis respond: returns
HTTP 200 OK. - If either dependency is unreachable or times out: returns
HTTP 503 Service Unavailable.
Active Deployment Health Inspector (GET /admin/v1/health/deployments)
Provides the active health status across all configured upstream model deployments:
curl -X GET "http://localhost:8080/admin/v1/health/deployments" \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me"
Response:
{
"data": [
{
"deployment_id": "dep_openai_gpt4o",
"deployment_name": "OpenAI US-East Production",
"provider": "openai",
"model": "gpt-4o",
"status": "healthy",
"consecutive_failures": 0,
"probe_latency_ms": 142,
"last_checked_at": "2026-11-11T16:00:00Z"
},
{
"deployment_id": "dep_anthropic_sonnet",
"deployment_name": "Anthropic Direct",
"provider": "anthropic",
"model": "claude-3-5-sonnet",
"status": "degraded",
"consecutive_failures": 1,
"probe_latency_ms": 1250,
"last_error": "rate_limit: 429 Too Many Requests",
"last_checked_at": "2026-11-11T16:00:00Z"
}
]
}
Triggering Instant Active Sweep (POST /admin/v1/health/probe)
Operators can force an immediate, non-blocking health probe sweep across all registered deployments (testing credentials and upstream connectivity via ListModels or ping):
curl -X POST "http://localhost:8080/admin/v1/health/probe" \
-H "Authorization: Bearer sk-gw-master-admin-default-key-change-me"
The gateway audits the trigger action in audit_logs under action health.deployments.probed and updates all deployment circuit breaker states accordingly.
4. Kubernetes Production Manifest Example
Configure your Kubernetes deployment pod spec with tailored probes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: goroute-ai-gateway
spec:
replicas: 3
template:
spec:
containers:
- name: gateway
image: ghcr.io/goroute-ai/ai-gateway-go:v1.8.0
ports:
- containerPort: 8080
name: http
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
httpGet:
path: /readyz
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 2