Skip to main content

Client Initialization & Configuration

The pkg/gateway package uses functional options to configure an embedded gateway or a remote gateway client.


1. Initializing the Client​

gateway.New(opts ...Option) (*Client, error)​

Initializes a client. If WithRemote(...) is provided, it connects to that remote service. Otherwise, it loads configuration and starts an embedded gateway server in the current process:

gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{ModelName: "fast", Model: "gemini/gemini-2.5-flash"},
),
gateway.WithLogLevel("info"),
)
if err != nil {
log.Fatalf("failed to initialize: %v", err)
}
defer gw.Close()

gateway.NewClient(baseURL, apiKey string) *Client​

Creates an HTTP client pointing to an already-running GoRoute server or container:

client := gateway.NewClient("http://localhost:8080", "sk-gw-developer-key")

2. Configuration Options​

WithModelList(entries ...ModelEntry) Option​

Configures the embedded router with declarative model aliases, fallbacks, and routing strategies.

gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{
ModelName: "smart-router",
Model: "gemini/gemini-2.5-flash",
APIKey: "os.environ/GEMINI_API_KEY",
RoutingStrategy: "least-busy",
Fallbacks: []string{"backup-claude"},
RPM: 1000,
TPM: 500000,
},
gateway.ModelEntry{
ModelName: "backup-claude",
Model: "anthropic/claude-sonnet-4-20250514",
APIKey: "os.environ/ANTHROPIC_API_KEY",
},
),
)

ModelEntry Specification​

FieldTypeDescription
ModelNamestringClient-facing alias (e.g. "fast", "reasoning", "auto").
ModelstringUpstream provider/model identifier (e.g. "gemini/gemini-2.5-flash", "anthropic/claude-sonnet-4-20250514", "openai/gpt-4.1-mini").
APIKeystringUpstream API key string or environment reference (e.g. "os.environ/GEMINI_API_KEY").
APIBasestringOptional custom endpoint URL (e.g. for Ollama or private vLLM clusters: "http://localhost:11434").
RoutingStrategystringRouting policy: "weighted-shuffle", "least-busy", "latency-based", "usage-based-headroom", "lowest-cost", or "round-robin".
Fallbacks[]stringOrdered list of model aliases to invoke if this model returns 429, 503, or times out.
WeightintInteger weight for "weighted-shuffle" distribution.
RPMintRequests Per Minute limit enforced across this model entry.
TPMintTokens Per Minute limit enforced across this model entry.
SimulateStatusintOptional fault injection HTTP code (e.g. 503) to test circuit breakers and fallback chains offline.

WithConfigFile(path string) Option​

Loads model definitions, rate limits, and security policies from an external YAML file:

gw, err := gateway.New(
gateway.WithConfigFile("./config/gateway.yaml"),
)

Sample YAML (gateway.yaml):

model_list:
- model_name: "production-fast"
model_params:
model: "gemini/gemini-2.5-flash"
api_key: "os.environ/GEMINI_API_KEY"
rpm: 1000
fallbacks: ["production-backup"]
- model_name: "production-backup"
model_params:
model: "openai/gpt-4.1-mini"
api_key: "os.environ/OPENAI_API_KEY"

WithDemoMocksIfEnvMissing() Option​

Enables built-in local mock HTTP servers for any ModelEntry whose environment variable API key is not present. This allows unit tests, integration tests, and local demos to run immediately without requiring live provider credentials or incurring costs.

gw, err := gateway.New(
gateway.WithModelList(
gateway.ModelEntry{ModelName: "test-model", Model: "gemini/gemini-2.5-flash"},
),
gateway.WithDemoMocksIfEnvMissing(), // Mock provider handles requests locally
)

Logging Options​

Control internal gateway logging using Go standard library's log/slog:

  • WithLogger(l *slog.Logger) Option: Provide a custom slog.Logger instance.
  • WithSilentLogs() Option: Suppress all stdout/stderr logs (ideal for CLI tools or clean terminal outputs).
  • WithLogLevel(level string) Option: Set minimum log level ("debug", "info", "warn", "error").
customLogger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelDebug}))

gw, err := gateway.New(
gateway.WithLogger(customLogger),
)

WithRemote(baseURL, apiKey string) Option​

Redirects gateway.New to connect to a remote gateway daemon instead of starting an embedded router:

gw, err := gateway.New(
gateway.WithRemote("https://ai-gateway.company.com", "sk-gw-production"),
)

WithHTTPClient(hc *http.Client) Option​

Configures a custom *http.Client for remote gateway HTTP requests and /v1/realtime WebSocket dialing (timeouts, custom TLS, transport pooling):

customHTTP := &http.Client{Timeout: 45 * time.Second}
gw, err := gateway.New(
gateway.WithRemote("https://ai-gateway.company.com", "sk-gw-production"),
gateway.WithHTTPClient(customHTTP),
)

WithAllowPrivateNetworks() Option​

Permits embedded gateway connections to localhost or private network addresses (useful for local Ollama/LM Studio servers or in-process test mocks without enabling WithDemoMocksIfEnvMissing()).


3. Client Lifecycle & Scoping​

Dynamic Key Switching with WithKey​

In multi-tenant backend services, create a shallow copy of the client with a tenant's specific Virtual API Key. The base connection and connection pooling are shared:

// Base client initialized once at service startup
masterClient, _ := gateway.New(...)

// HTTP handler for tenant A
func TenantAHandler(w http.ResponseWriter, r *http.Request) {
tenantKey := r.Header.Get("X-Tenant-Key")
tenantClient := masterClient.WithKey(tenantKey)

// Chat requests are now governed by tenantKey's budget, rate limits, and model allowances
res, err := tenantClient.Chat(r.Context(), "fast", "Process tenant query")
...
}

Teardown with Close​

For embedded instances, Close() cleanly shuts down background workers, flushes remaining telemetry events, and terminates local mock servers:

defer gw.Close()

(For remote clients created via NewClient, Close() is a safe no-op).

Endpoint Inspection with BaseURL​

Inspect the loopback HTTP URL generated for the in-process server (e.g. http://127.0.0.1:54321):

fmt.Printf("Embedded gateway listening internally on: %s\n", gw.BaseURL())