Skip to main content

Asynchronous Batch Processing API

For high-volume, non-interactive workloads (such as dataset labeling, offline evaluation, bulk summarization, and synthetic data generation), GoRoute AI Gateway provides the OpenAI-Compatible Batch Processing API.

Batches decouple request ingestion from immediate execution. By accepting a 24-hour completion SLA, workloads run over off-peak provider headroom at up to a 50% discount compared to synchronous on-demand inference.


Endpoint Specification​

EndpointMethodOutputDescription
/v1/batchesPOSTCanonicalBatch JSONSubmit a new asynchronous batch job.
/v1/batchesGETList JSONList all recent batches for the organization.
/v1/batches/{batch_id}GETCanonicalBatch JSONInspect execution status and progress counts.
/v1/batches/{batch_id}/cancelPOSTCanonicalBatch JSONAbort an ongoing or pending batch job.
  • Authentication: Bearer sk-gw-... (Virtual Key or Master Admin Key)
  • Turnaround SLA (completion_window): 24h
  • Supported Target Endpoint: /v1/chat/completions

State Machine Transitions​

POST /v1/batches Valid input file & schema Invalid file or syntax error All records processed POST /v1/batches/{id}/cancel Disk or database failure validating in_progress failed completed cancelled
StateDescription
validatingGateway checks tenant ownership of input_file_id, file purpose (batch), and verifies initial JSONL syntax.
in_progressBackground worker parses records sequentially, checks RBAC permissions per line item, and dispatches inference.
completedAll items have completed; output JSONL file is finalized and registered with purpose: "batch_output".
failedThe batch encountered an unrecoverable failure (e.g. malformed lines exceeding buffer limits or storage loss).
cancelledThe caller explicitly aborted the batch before completion; completed items are preserved in output.

JSONL File Formats​

1. Input File (.jsonl)​

Upload using POST /v1/files with purpose="batch":

{"custom_id": "eval-task-001", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "fast", "messages": [{"role": "user", "content": "Classify sentiment: Great latency."}]}}
{"custom_id": "eval-task-002", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "fast", "messages": [{"role": "user", "content": "Classify sentiment: Connection dropped."}]}}

2. Output File (.jsonl)​

Once the batch reaches status: "completed", download via GET /v1/files/{output_file_id}/content:

{"id": "batch_req_01a", "custom_id": "eval-task-001", "response": {"status_code": 200, "request_id": "req_88f9...", "body": {"choices": [{"message": {"role": "assistant", "content": "Positive"}}]}}, "error": null}
{"id": "batch_req_02b", "custom_id": "eval-task-002", "response": {"status_code": 200, "request_id": "req_11a4...", "body": {"choices": [{"message": {"role": "assistant", "content": "Negative"}}]}}, "error": null}

Code Examples​

# 1. Upload input file
FILE_ID=$(curl -s -X POST http://localhost:8080/v1/files \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-F "file=@batch_tasks.jsonl" \
-F "purpose=batch" | jq -r .id)

# 2. Submit Batch job
BATCH_ID=$(curl -s -X POST http://localhost:8080/v1/batches \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d "{
\"input_file_id\": \"$FILE_ID\",
\"endpoint\": \"/v1/chat/completions\",
\"completion_window\": \"24h\",
\"metadata\": {\"job_name\": \"nightly-evals\"}
}" | jq -r .id)

echo "Created batch: $BATCH_ID"

# 3. Check Batch Status
curl -X GET "http://localhost:8080/v1/batches/$BATCH_ID" \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"

# 4. Cancel Batch (if needed)
curl -X POST "http://localhost:8080/v1/batches/$BATCH_ID/cancel" \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e"

FinOps & Cost Economics​

Because batch execution allows upstream providers to schedule processing during lower-demand periods, GoRoute applies the batch modality rate:

  • Up to 50% discount relative to synchronous inference rates.
  • Each completed request line item commits an atomic SpendEvent with modality: "batch" and exact token counts.
  • Failed items do not incur completion token costs.