Skip to main content

Chat and Streaming

The SDK exposes Chat for a single user prompt, ChatWithMessages for structured history, and StreamChat for receiving text deltas through a callback. The examples below are function-body snippets and assume ctx context.Context, an initialized gw, and the standard imports.

Single-prompt chat​

result, err := gw.Chat(ctx, "fast", "Draft a short status update.",
gateway.WithSystem("You are a concise engineering assistant."),
gateway.WithTemperature(0.4),
gateway.WithMaxTokens(256),
)
if err != nil {
return err
}
fmt.Println(result.Content)

Available request options are WithSystem, WithTemperature, WithMaxTokens, and WithReasoningEffort. The upstream provider determines which options it supports.

Multi-turn chat​

result, err := gw.ChatWithMessages(ctx, gateway.ChatRequest{
Model: "fast",
Messages: []gateway.Message{
{Role: "system", Content: "You are a concise site reliability engineer."},
{Role: "user", Content: "What should I check first after an HTTP 504?"},
{Role: "assistant", Content: "Check the gateway, upstream health, and timeout logs."},
{Role: "user", Content: "How can I narrow it down?"},
},
})
if err != nil {
return err
}
fmt.Println(result.Content)

Stream text deltas​

StreamChat calls onDelta as text arrives from the upstream stream. The callback can return an error to stop processing.

result, err := gw.StreamChat(ctx, "fast", "Write a short haiku about Go.", func(delta string) error {
_, err := fmt.Print(delta)
return err
})
if err != nil {
return err
}
fmt.Printf("\nCost: %d µ$\n", result.CostMicros)

The gateway translates supported provider streaming responses into the SDK callback. Delivery timing and available content depend on the selected provider; no fixed latency or buffering guarantee is implied.

Set a deadline​

Pass a context with a timeout to bound the request and cancel work when the caller no longer needs the response:

requestCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()

result, err := gw.Chat(requestCtx, "fast", "Summarize this report.")
if err != nil {
return err
}
fmt.Println(result.Content)

Pure-SDK Modality & Responses Wrappers​

In v1.7.0, pkg/gateway provides self-contained public request and response structs (zero internal/ package imports) for embeddings, reranking, moderation, the OpenAI Responses API, images, audio, and tokenization:

// Vector embeddings
emb, err := gw.CreateEmbedding(ctx, gateway.EmbeddingRequest{
Model: "text-embedding-3-small",
Input: []string{"GoRoute AI Gateway"},
})

// Document reranking
reranked, err := gw.CreateRerank(ctx, gateway.RerankRequest{
Model: "rerank-v3.5",
Query: "zero-allocation Go proxy",
Documents: []any{"GoRoute routes inference in pure Go."},
})

// OpenAI Responses API (sync or streaming)
resp, err := gw.CreateResponse(ctx, gateway.ResponseRequest{
Model: "fast",
Input: "Summarize this diff in two bullets.",
})
_, err = gw.StreamResponse(ctx, gateway.ResponseRequest{
Model: "fast",
Input: "Stream a haiku about Go.",
}, func(delta string) error {
_, err := fmt.Print(delta)
return err
})

Additional pure-SDK modality methods include CreateModeration, CreateImage, ImageEdit, ImageVariation, CreateSpeech, TranscribeAudio, Tokenize, CountTokens, and stateful file/batch helpers (UploadFile, ListFiles, GetFile, GetFileContent, DeleteFile, CreateBatch, ListBatches, GetBatch, CancelBatch).

Realtime WebSocket Sessions​

Use ConnectRealtime to open a full-duplex WebSocket session to /v1/realtime using *gateway.RealtimeSession:

session, err := gw.ConnectRealtime(ctx, "gpt-4o-realtime-preview")
if err != nil {
return err
}
defer session.Close()

if err := session.SendEvent(ctx, gateway.RealtimeEvent{
Type: "response.create",
}); err != nil {
return err
}

ev, err := session.ReadEvent(ctx)
if err != nil {
return err
}
fmt.Println("Received event:", ev.Type)

Inspect result fields​

ChatResult provides top-level fields for content, provider, model, request ID, cost in micro-dollars, routing time in microseconds, cache status, and whether a fallback was attempted. Usage and Raw expose internal domain types; prefer the top-level fields when you need a stable public SDK surface.

For function/tool definitions and the HTTP request shape, see Tool Calling. The current public ChatRequest struct does not expose a tools field.