Skip to main content

Troubleshooting & Recovery Runbooks

This guide provides root-cause diagnostic workflows and operational recovery runbooks for common production failure modes in GoRoute AI Gateway.

HTTP 402 HTTP 429 HTTP 403 / 502 SSRF HTTP 502 / 503 HTTP 401 / 403 Auth Client Encounters Error Response Check HTTP Status & Error Code Budget Exceeded (budget_exceeded_error)→ Runbook 1: Adjust Spend Limits Rate Limited (rate_limit_error)→ Runbook 2: Sliding-Window Quota / Leases SSRF or Private Network Block→ Runbook 3: Network Policy & CIDRs Circuit Breaker Tripped (OPEN)→ Runbook 4: Cooldown & Probing Auth / Permission Error→ Runbook 5: 4-Layer RBAC & Keys

1. Unified Error Taxonomy Reference​

All data plane errors conform to standard OpenAI error formats with machine-readable type and code properties:

{
"error": {
"type": "rate_limit_error",
"code": "virtual_key_rpm_limit_exceeded",
"message": "Virtual key sk-gw-a1b2 has exceeded its limit of 100 requests per minute."
}
}
HTTP StatusError typeTypical Error codeRoot Cause
400 Bad Requestinvalid_request_errorinvalid_json, missing_modelRequest body syntax error or unrecognized parameters.
401 Unauthorizedauthentication_errorinvalid_api_key, key_expiredMissing, malformed, or expired client virtual key.
402 Payment Requiredbudget_exceeded_errororg_budget_exceeded, key_budget_exceededHard micro-dollar spending limit reached by tenant or key.
403 Forbiddenpermission_errormodel_forbidden, mcp_forbidden4-layer RBAC policy prevents key from accessing target model or server.
404 Not Foundnot_found_errormodel_not_found, task_not_foundRequested virtual alias or A2A task does not exist.
429 Too Many Requestsrate_limit_errorrate_limit_rpm_exceeded, concurrency_limitSliding-window RPM/TPM limit or concurrent in-flight lease limit reached.
502 Bad Gatewayupstream_provider_errorupstream_auth_error, upstream_bad_gatewayUpstream provider rejected credential or failed with 5xx.
503 Service Unavailableupstream_provider_errorcircuit_open, all_deployments_downCircuit breaker tripped; all primary and fallback deployments offline.
504 Gateway Timeoutupstream_provider_errorupstream_timeoutUpstream provider exceeded write/read timeout.

2. Runbook 1: Financial Spend Budget Exceeded (HTTP 402)​

Symptom​

Client receives HTTP 402 with error type budget_exceeded_error:

{
"error": {
"type": "budget_exceeded_error",
"code": "org_budget_exceeded",
"message": "Configured financial spend budget exceeded (org_budget_exceeded)"
}
}

Diagnosis​

  1. Inspect the caller's virtual key and tenant association:
curl -X GET "http://localhost:8080/admin/v1/virtual-keys?prefix=sk-gw-a1b2" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
  1. Check current accumulated spend vs maximum budget in PostgreSQL:
SELECT id, name, spent_micros, max_budget_micros,
(spent_micros::float / NULLIF(max_budget_micros, 0) * 100) AS pct_used
FROM organizations
WHERE slug = 'acme-corp';

Remediation​

  • Option A: Increase Organization or Key Budget via Admin API:
curl -X PATCH "http://localhost:8080/admin/v1/organizations/org_12345" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
-H "Content-Type: application/json" \
-d '{"max_budget_micros": 500000000}' # $500.00 USD
  • Option B: Reset Accumulated Spend for a New Billing Cycle:
UPDATE organizations
SET spent_micros = 0, budget_reset_at = NOW()
WHERE id = 'org_12345';

The gateway will automatically clear the Redis spend accumulator key (gw:spend:org:org_12345).


3. Runbook 2: Rate Limit or Concurrency Rejection (HTTP 429)​

Symptom​

Requests fail with HTTP 429. The response contains a Retry-After: <seconds> header.

Diagnosis​

  1. Check gateway logs for rate limiter evaluation messages:
grep "rate_limit_exceeded" /var/log/goroute/gateway.log
  1. Check Redis sliding-window counters:
redis-cli KEYS "gw:rl:*:sk-gw-a1b2:*"
redis-cli GET "gw:rl:conc:sk-gw-a1b2"

Remediation​

  • If Requests Per Minute (RPM) or Tokens Per Minute (TPM) are exhausted: Update the virtual key limits:
    curl -X PATCH "http://localhost:8080/admin/v1/virtual-keys/key_12345" \
    -H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY" \
    -H "Content-Type: application/json" \
    -d '{"rpm_limit": 500, "tpm_limit": 100000}'
  • If Concurrency Locks are Stale: In rare scenarios where client connections abort without cleanly releasing concurrent lease counters, expire the concurrency key:
    redis-cli DEL "gw:rl:conc:key_12345"

4. Runbook 3: Outbound SSRF & Private Network Blocks​

Symptom​

Requests to self-hosted models (vLLM, Ollama, TGI) or private VPC endpoints fail with errors such as:

[502:upstream_error] network: private/local network address is forbidden by SSRF policy

Or:

network: destination IP is in the forbidden list

Root Cause​

The gateway enforces a zero-trust outbound security policy (internal/security/network/policy.go):

  • Prohibits loopback (127.0.0.0/8, ::1/128).
  • Prohibits RFC 1918 private subnets (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).
  • Prohibits cloud metadata endpoints (169.254.169.254, fd00:ec2::254).
  • Blocks plaintext http:// schemes when allow_insecure_http is false.

Remediation for Internal Models​

If you are hosting private inference engines within a secure VPC, configure explicit CIDR allowlists or enable private network routing in configs/config.yaml:

security:
network:
outbound_ssrf:
allow_private_networks: true
allowed_schemes:
- "https"
- "http"
allowed_private_cidrs:
- "10.240.0.0/16" # Your internal Kubernetes pod CIDR
- "192.168.1.0/24" # On-premise GPU cluster

[!CAUTION] Never configure 169.254.169.254 or 0.0.0.0/0 in allowed_private_cidrs. Cloud instance metadata must remain permanently blocked to prevent credential theft.


5. Runbook 4: Tripped Circuit Breakers (CircuitOpen)​

Symptom​

Inference requests fail immediately with HTTP 503 without reaching the upstream provider:

{
"error": {
"type": "upstream_provider_error",
"code": "circuit_open",
"message": "Upstream deployment dep_openai_east is isolated in circuit breaker cooldown"
}
}

Diagnosis​

  1. Inspect deployment statuses:
curl -X GET "http://localhost:8080/admin/v1/health/deployments" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
  1. Verify if the deployment is in cooldown in Redis:
redis-cli GET "gw:router:cooldown:dep_openai_east"

Remediation​

  • Wait for Cooldown Expiration: By default, the deployment remains isolated for router.default_cooldown (60 seconds). Once expired, the circuit enters HALF_OPEN state, allowing a single trial probe to verify recovery.
  • Force Immediate Recovery via Active Probe:
curl -X POST "http://localhost:8080/admin/v1/health/probe" \
-H "Authorization: Bearer $GATEWAY_MASTER_ADMIN_KEY"
  • Manual Reset: Delete the Redis cooldown lock directly:
redis-cli DEL "gw:router:cooldown:dep_openai_east"

6. Runbook 5: 4-Layer RBAC Authorization Failures (HTTP 403)​

Symptom​

Client receives HTTP 403:

{
"error": {
"type": "permission_error",
"code": "model_forbidden",
"message": "Virtual key sk-gw-a1b2 is not authorized to use model \"o3-mini\""
}
}

Diagnosis: Trace the 4-Layer Hierarchy​

The gateway checks 4 hierarchical layers in order. If any layer denies access, the request is rejected fail-closed:

Yes No Yes No Yes No Yes No Request for 'o3-mini' Layer 1: Global CatalogueIs model active? Layer 2: Org SubscriptionIs provider enabled for org? 403 Forbidden Layer 3: Org Model PolicyIs model approved? Layer 4: Virtual Key AllowlistIs model in allowed_models? 200 OK (Allowed)

Remediation Queries​

  1. Verify Layer 2 Subscription:
SELECT * FROM org_provider_subscriptions
WHERE org_id = 'org_default' AND provider = 'openai';
  1. Verify Layer 3 Model Approval:
SELECT * FROM org_model_policies
WHERE org_id = 'org_default' AND global_model_id = 'openai/o3-mini';
  1. Verify Layer 4 Key Allowlists:
SELECT allowed_models FROM virtual_keys WHERE key_prefix = 'sk-gw-a1b2';

Update the key allowlist to include the requested model or "*".