Pre-Flight Tokenizer
To enforce Tokens Per Minute (TPM) quotas before dispatching expensive upstream requests, GoRoute AI Gateway runs a fast pre-flight tokenizer.
1. Why Pre-Flight Tokenization Matters
Without pre-flight token estimation, a client could submit a massive prompt (e.g. 100k tokens) that consumes your entire upstream quota before the gateway can register the spend. By estimating token volume before sending the request upstream, GoRoute reserves tokens in Redis and rejects oversized requests with HTTP 429 before incurring upstream billing.
2. Token Estimation Heuristics
The pre-flight tokenizer balances speed and accuracy using a zero-allocation scanner:
- Exact BPE Encoding: For OpenAI and Mistral models, uses Byte-Pair Encoding token tables.
- Sub-Word Heuristic Fast Path: For high-throughput scenarios (< 5 µs target overhead), computes token approximations based on character classes, punctuation density, and whitespace boundaries:
Estimated Tokens = (Byte Count / 4) * Complexity Multiplier
If the estimated prompt tokens + requested max_tokens exceeds the virtual key's available TPM quota, the request is throttled immediately.