Skip to main content

Fast Heuristic Tokenization API

Accurate token estimation is vital for client-side budgeting, chunking text for RAG (Retrieval-Augmented Generation), and pre-flight validation before executing long-context inference.

GoRoute AI Gateway includes a zero-latency, in-process subword tokenizer accessible via POST /v1/tokenize. Unlike traditional proxies that make network roundtrips to an external tokenizer service or invoke heavyweight Cgo Python bindings (like tiktoken), GoRoute runs an optimized, pure-Go heuristic scanner that approximates Byte Pair Encoding (BPE) and SentencePiece vocabularies in under 50 microseconds.


Endpoint Specification​

  • HTTP Method: POST
  • Supported Paths:
    • /v1/tokenize
    • /tokenize (Unversioned alias)
  • Authentication: Bearer sk-gw-... (Virtual Key or Master Admin Key)
  • Content-Type: application/json

Tokenizer Profiles & Vocabulary Heuristics​

The gateway dynamically resolves models to one of five specialized tokenization profiles (internal/ratelimit/tokenizer.go):

ProfileApplicable ModelsVocabulary DivisorDescription
o200k_baseGPT-4o, GPT-4.1, o1, o3, o44.2 chars/token200k vocabulary; high compression on multilingual & code.
cl100k_baseGPT-4, GPT-3.5, text-embedding-3-*3.9 chars/tokenStandard 100k OpenAI BPE vocabulary.
claude_bpeClaude 3.5 Sonnet, Claude 3.7, Claude 3 Opus3.6 chars/tokenDenser subword encoding on prose and system instructions.
gemini_spGemini 2.5 Flash, Gemini 2.5 Pro, Gemma4.1 chars/tokenGoogle SentencePiece unigram/BPE mixture.
mistral_tekkenMistral Large, Codestral, Llama 3/3.33.8 chars/tokenTekken tokenizer with expanded code character tables.

Mathematical Estimation Algorithm​

For any given input text, the scanner decomposes the byte stream across three dimensions:

Tokens = ⌈(AlnumChars * 100) / Divisor⌉ + ⌊(PunctChars + 1) / 2⌋ + ⌊(UTF8NonASCIIBytes + 1) / 2⌋
  1. Alphanumeric Characters: Scaled using the family-specific character-to-token ratio.
  2. Punctuation & Code Syntax: Code brackets ({, }, [, ]), JSON quotes, and operators often form dedicated BPE tokens.
  3. Multi-Byte UTF-8 Bytes: Multi-byte sequences (emojis, Cyrillic, CJK, Arabic) undergo multi-step BPE merges.

Request & Response Schemas​

Request Schema​

Supports either a raw text prompt, or structured chat messages and tool definitions:

{
"model": "gpt-4.1",
"prompt": "Explain Byzantine fault tolerance in distributed databases."
}

Or for structured multi-turn conversations:

{
"model": "claude-3-7-sonnet",
"messages": [
{"role": "system", "content": "You are a concise code reviewer."},
{"role": "user", "content": "Review this pull request diff."}
],
"tools": [
{
"type": "function",
"function": {
"name": "lookup_git_diff",
"description": "Fetch PR diff by commit hash",
"parameters": {
"type": "object",
"properties": {
"commit": {"type": "string"}
}
}
}
}
]
}

Response Schema​

{
"model": "gpt-4.1",
"profile": "o200k_base",
"count": 9,
"total_tokens": 9
}

Code Examples​

# Tokenize a raw prompt
curl -X POST http://localhost:8080/v1/tokenize \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4.1",
"prompt": "GoRoute AI Gateway performs zero-allocation subword tokenization."
}'

# Tokenize full chat messages using Claude BPE profile
curl -X POST http://localhost:8080/v1/tokenize \
-H "Authorization: Bearer sk-gw-prod-7f9a2b8c4d1e" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-3-5-sonnet",
"messages": [
{"role": "system", "content": "You are an AI assistant."},
{"role": "user", "content": "How do goroutines manage call frames?"}
]
}'

Comparison: In-Process Heuristic vs Network Tokenizer​

FeatureGoRoute /v1/tokenizeRemote Tokenizer ServiceCgo / Tiktoken Binding
Execution Latency< 0.05 ms (in-process)15 – 50 ms (HTTP network)0.5 – 2 ms
Memory AllocationZero heap allocationTCP socket buffersCgo stack transition overhead
Network Failure ModeNone (always available)External point of failureProcess segmentation risk
Cross-Provider ProfilesAuto-adapts to GPT, Claude, Gemini, MistralUsually single vendor onlyRequires multiple native libraries

[!NOTE] The in-process token counter is also invoked automatically on every incoming /v1/chat/completions request for pre-flight rate limiting and sliding-window quota reservations.