Skip to main content

AI FinOps: Cost Management & Optimization

When several applications and teams use AI, it can be difficult to see where spend comes from or keep usage within limits. GoRoute records request costs, checks budgets before sending work to a provider, and can enforce request and token limits. The detailed guides explain how this accounting works and what it means for your setup.


Core Capabilities​

CapabilityImplementationGuide
64-Bit Integer Micro-Dollar LedgerAll costs are computed in integer micro-dollars (1 USD = 1,000,000 micro-dollars) with intermediate pico-dollar precision and zero float32/float64 accounting.Micro-Dollar Ledger
Cache-Write & Modality PricingSupports prompt-cache write pricing (CacheCreationTokens / cache_write_cost_micros_per_1m), audio/image/video modality rates (0007/0012), and immutable model_pricing_history.Micro-Dollar Ledger
Unpriced Usage TrackingTracks unpriced_requests and unpriced_tokens explicitly so custom or uncatalogued models never silently report $0.00 without visibility.Micro-Dollar Ledger
Pre-Flight Reservations & Lease RenewalReserves upper-bound estimated cost before upstream dispatch, renews leases during long streams, and reconciles exact token spend on completion.Micro-Dollar Ledger
Durable PostgreSQL OutboxesPersistent spend (0008) and budget-reset (0011) outbox tables ensure zero lost accounting events across restarts or transient DB blips.Deployment & Storage
Single-Roundtrip Redis Lua Rate LimitsAtomic sliding-window RPM and TPM enforcement in Redis with fail-closed behavior and {org:<id>} hash-tag compatibility for Redis Cluster.Sliding-Window Rate Limits
Pre-Flight TokenizerLocal zero-network token estimation (POST /v1/tokenize, POST /anthropic/v1/messages/count_tokens) to enforce TPM/context budgets before dispatch.Pre-Flight Tokenizer

FinOps Accounting Lifecycle​

Inference Request (sk-gw-...) Pre-Flight Tokenizer (Estimate Input Tokens) Atomic Lua Check (RPM / TPM + Upper-Bound Budget Lease) Dispatch Inference (Renew Lease if Streaming) Usage Totals (Input, Output, Cache Creation) Reconcile Exact Micro-Dollar Spend Append Spend Event & Immutable Pricing Audit Client Gateway Redis (Lua + Leases) PostgreSQL (Outbox + Ledger) Upstream

Explore FinOps Guides​