Skip to main content

OpenAI Realtime WebSocket Protocol

GoRoute AI Gateway supports the OpenAI Realtime API protocol over bidirectional WebSockets. Realtime enables low-latency, full-duplex conversational sessions supporting streaming text and audio with sub-second voice interactions.


Endpoint Specification​

  • HTTP Method: GET (WebSocket Upgrade Handshake)
  • Supported Paths:
    • /v1/realtime (Standard data plane endpoint)
    • /realtime (Unversioned compatibility alias)
    • /openai/v1/realtime (OpenAI wire compatibility alias)
  • Subprotocol: realtime (passed via Sec-WebSocket-Protocol: realtime)
  • Authentication: Bearer sk-gw-... passed in the Authorization header during the WebSocket upgrade handshake
  • Default Model: gpt-4o-realtime-preview (configurable via ?model=... query parameter)

Architecture: Zero-Allocation Connection Hijacking​

GoRoute utilizes nhooyr.io/websocket for RFC 6455 compliance without external daemon processes. When a client initiates the handshake:

loop [Full-Duplex Stream Loop] 1. GET /v1/realtime (WebSocket Upgrade) 2. HTTP 101 Switching Protocols 3. ServerEvent: session.created 4. ClientEvent: response.create 5. Stream Realtime Audio Frames 6. Raw Tokens / Audio Deltas 7. ServerEvent: audio.delta 8. WebSocket Close Authenticate Key & Model RBAC Hijack TCP Connection Enqueue SpendEvent to FinOps Ledger Client GoRoute AI Gateway Upstream Realtime Engine
  1. In-Flight Authentication: Handshake requests pass through auth.DataPlaneMiddleware, verifying the virtual key and validating model access permissions before the TCP connection is hijacked.
  2. Session State Isolation: Each active WebSocket maintains its own isolated message history, active tools, and audio buffer frames.
  3. Session FinOps Ledger: When the session closes, the gateway aggregates all tokens and audio duration generated throughout the session and records a single consolidated spend transaction.

Realtime Event Protocol​

Communication occurs via JSON-formatted frames over text message blocks.

1. Client Events (Inbound)​

Event TypeDescription
session.updateModifies active session parameters (modalities, voice, temperature, instructions).
conversation.item.createAppends a new user message or tool result to conversation history.
input_audio_buffer.appendAppends base64-encoded PCM16 audio bytes to the input buffer.
input_audio_buffer.commitCommits pending audio and triggers automatic turn detection.
response.createInstructs the model to generate an assistant turn immediately.
response.cancelInterrupts current streaming generation (e.g. user barge-in).

2. Server Events (Outbound)​

Event TypeDescription
session.createdEmitted immediately upon connection with active session configuration.
session.updatedAcknowledges receipt and application of session.update.
conversation.item.createdConfirms item addition to session context.
response.createdSignals that a new response generation run has begun.
response.text.deltaIncremental text token delta from the assistant.
response.audio.deltaIncremental base64-encoded PCM16 audio chunks.
response.doneSignals completion of the response turn with final usage metadata.
errorEmitted if an invalid event, schema mismatch, or budget error occurs.

Session Configuration (session.created)​

Immediately upon upgrading the connection, GoRoute dispatches the initial session configuration:

{
"event_id": "event_88fa2b10-2180-4ce9-9f01-38290fa1b203",
"type": "session.created",
"session": {
"id": "sess_1189ac02-990a-412e-b998-382910fa8821",
"object": "realtime.session",
"model": "gpt-4o-realtime-preview",
"modalities": ["text"],
"instructions": "You are a helpful AI assistant.",
"voice": "alloy",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16"
}
}

Code Examples​

import asyncio
import json
import websockets

GATEWAY_WS_URL = "ws://localhost:8080/v1/realtime?model=gpt-4o-realtime-preview"
API_KEY = "sk-gw-prod-7f9a2b8c4d1e"

async def realtime_chat():
headers = {"Authorization": f"Bearer {API_KEY}"}
async with websockets.connect(GATEWAY_WS_URL, extra_headers=headers, subprotocols=["realtime"]) as ws:
# 1. Wait for session.created
init_event = json.loads(await ws.recv())
print(f"Session established: {init_event['session']['id']}")

# 2. Send user text message
await ws.send(json.dumps({
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [{"type": "input_text", "text": "What is connection hijacking in HTTP servers?"}]
}
}))

# 3. Request response generation
await ws.send(json.dumps({"type": "response.create"}))

# 4. Stream response events
while True:
msg = json.loads(await ws.recv())
event_type = msg.get("type")
if event_type == "response.text.delta":
print(msg.get("delta", ""), end="", flush=True)
elif event_type == "response.done":
print("\n[Turn completed]")
break

asyncio.run(realtime_chat())