Tokenizer
The program that chops text into tokens before the model sees it.
The fixed set of rules mapping text to token IDs and back. Every model has its own tokenizer, so the same string can cost different amounts on different models.
Tokenizers are why models are bad at counting letters and why some languages and code symbols cost more per word. Rarer scripts and unusual punctuation fragment into many tokens.
The practical consequence is that token counts are model-specific, so the same prompt costs different amounts on different providers and in different languages. Code, JSON and non-English text tokenise far less efficiently than English prose, which is a real cost multiplier on developer-facing products.
The language that doubled the bill
A product ships in English, then adds a language whose script tokenises poorly.
The same content needs noticeably more tokens, so cost and context usage both rise with no change in behaviour. Worth measuring per language before promising a multilingual rollout at the same unit price.
- docsGet token count ↗
How text and multimodal input are tokenised and counted, with usage reporting. Concrete counts are model-specific; the concept is not.
- guideIntroduction to large language models ↗
A plain-language official introduction to models, tokens and context. Teaching material rather than origin research — good for the definition, not for a claim about the frontier.
Token
The chunk of text the model actually sees — and the unit you are billed in.
Input Tokens · Prompt tokens
What you send. Cheaper, but repeated on every request.
Cost per Task
What one unit of real work actually costs.
LLM · Large Language Model
A very large model whose only job, at heart, is predicting the next token.