Skip to main content

Custom In-House Providers

GoRoute enables enterprises to onboard custom in-house inference clusters, private endpoints, and self-hosted model engines (such as vLLM, SGLang, TensorRT-LLM, TGI, and internal gateways) without writing custom Go code or recompiling the binary.

Custom providers are registered declaratively in configs/config.yaml through the certified OpenAI-compatible dialect subsystem (internal/adapter/openaicompat/registry.go:RegisterCustom).


Declarative Provider Registration (providers.custom_openai_compatible)​

To register a custom provider, declare it under the providers.custom_openai_compatible section of your configuration file:

providers:
custom_openai_compatible:
# 1. In-House vLLM GPU Cluster
- id: "internal-vllm"
display_name: "Internal vLLM H100 Cluster"
base_url: "https://vllm.internal.corp.net/v1"
supports_streaming: "true"
supports_tools: "true"
supports_structured_outputs: "true"
supports_list_models: true
default_headers:
X-Gateway-Identity: "goroute-prod"
X-Internal-Tenant: "engineering"

# 2. On-Premises TensorRT-LLM Engine
- id: "tensorrt-cluster"
display_name: "On-Premises TensorRT-LLM"
base_url: "http://10.200.4.15:8000/v1"
supports_streaming: "true"
supports_tools: "false"
supports_structured_outputs: "true"
supports_list_models: false
default_headers:
X-Acceleration-Engine: "TensorRT-LLM"

Configuration Attributes Reference​

FieldTypeRequiredDescription
idstringYesUnique identifier (must match ^[a-z0-9_\-]+$). Used as the model prefix (e.g. internal-vllm/llama-3.3-70b).
display_namestringNoHuman-readable description displayed in the Operator Web UI.
base_urlstringYesRoot HTTP/HTTPS endpoint of the inference server (including /v1).
supports_streamingstring / boolNoWhether the upstream supports Server-Sent Events (SSE). Defaults to true.
supports_toolsstring / boolNoWhether the upstream supports function and tool calling.
supports_structured_outputsstring / boolNoWhether the upstream supports response_format: {type: "json_object"}.
supports_list_modelsboolNoWhether GoRoute can query GET /models for auto-discovery.
default_headersmap[string]stringNoCustom HTTP headers injected into every request sent to this provider.

Registration Lifecycle & SSRF Validation​

When GoRoute parses custom provider entries during startup, it passes each profile through the ProfileRegistry.RegisterCustom validator:

Invalid Valid Not HTTP/HTTPS Valid Private IP / Prohibited CIDR Passed configs/config.yamlproviders.custom_openai_compatible Configuration Parser Validate ID^[a-z0-9_\\-]+$ Fatal: Invalid ID Validate Base URL & Scheme Fatal: Invalid Scheme SSRF Policy Verification(r.ssrfPolicy.ValidateURL) Fatal: Rejected by Outbound SSRF Policy Register into Dialect Profile Engine(New DialectProfile + Standard Strategies) Ready for Model List & Invocations

SSRF & Network Hardening for Internal Endpoints​

Because custom in-house clusters often reside on private corporate subnets (e.g., 10.0.0.0/8, 172.16.0.0/12), you must update GoRoute's outbound network policy:

security:
network:
outbound_ssrf:
allow_private_networks: true
allow_insecure_http: true # Required if internal cluster uses http:// instead of https://
allowed_schemes: ["http", "https"]
allowed_private_cidrs:
- "10.0.0.0/8"
- "172.16.0.0/12"
blocked_ips:
- "169.254.169.254" # Always block AWS/GCP cloud metadata IP
- "fd00:ec2::254"

[!CAUTION] Never disable cloud metadata blocking (169.254.169.254). GoRoute protects against SSRF vulnerabilities that could leak host credentials or cloud IAM tokens to unauthorized callers.


Configuring Models for Custom Providers​

Once registered, models from your custom provider can be referenced directly in model_list:

model_list:
# Using the registered custom provider ID as a prefix
- model_name: inhouse-llama
routing_strategy: weighted-shuffle
fallbacks: ["cloud-fallback"]
model_params:
model: internal-vllm/meta-llama/Llama-3.3-70B-Instruct
api_key: os.environ/INTERNAL_VLLM_API_KEY # Optional if cluster requires auth
weight: 100

# Direct ad-hoc override via api_base (without registering a custom provider)
- model_name: adhoc-sglang
model_params:
model: openai/meta-llama/Llama-3.1-8B-Instruct
api_base: "http://sglang-service.internal:30000/v1"
api_key: "dummy-key"

# Cloud backup fallback
- model_name: cloud-fallback
model_params:
model: groq/llama-3.3-70b-versatile
api_key: os.environ/GROQ_API_KEY

Usage Examples​

1. Invocations via cURL​

Send standard OpenAI requests referencing your custom model alias:

curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "inhouse-llama",
"messages": [
{"role": "system", "content": "You are a private corporate coding assistant."},
{"role": "user", "content": "Explain our internal microservice authentication flow."}
],
"temperature": 0.2
}'

2. Direct Dynamic Prefix Invocations​

If callers have permissions to direct-route, they can prefix the registered provider ID dynamically:

curl -X POST http://localhost:8080/v1/chat/completions \
-H "Authorization: Bearer your-gateway-virtual-key" \
-H "Content-Type: application/json" \
-d '{
"model": "internal-vllm/meta-llama/Llama-3.3-70B-Instruct",
"messages": [{"role": "user", "content": "Perform sanity check."}]
}'

3. Calling Custom In-House Providers via the Go SDK​

package main

import (
"context"
"fmt"
"log"

"github.com/goroute-ai/ai-gateway-go/pkg/gateway"
)

func main() {
ctx := context.Background()

gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "inhouse-llama",
Model: "openai/meta-llama/Llama-3.3-70B-Instruct",
APIBase: "https://vllm.internal.corp.net/v1",
APIKey: "os.environ/INTERNAL_VLLM_API_KEY",
},
),
)
if err != nil {
log.Fatalf("Failed to initialize gateway: %v", err)
}
defer gw.Close()

response, err := gw.Chat(ctx, "inhouse-llama", "Draft an internal architectural decision record (ADR).")
if err != nil {
log.Fatalf("Custom model chat error: %v", err)
}

fmt.Println("Custom Model Response:\n", response)
}

[!TIP] If your internal model engine serves streaming responses, GoRoute translates the stream into standard SSE without buffering, maintaining real-time Time-to-First-Token (TTFT) metrics for internal applications.