What is it?
Language models do not read letters or whole words directly. A tokenizer first converts text into a sequence of tokens — pieces drawn from a fixed vocabulary — and each token is mapped to a number. The model only ever sees and produces those numbers.
A token can be a whole common word (" the"), a fragment of a longer word ("un" + "believ" + "able"), a punctuation mark, a space, or a single character. Many tokenizers use a method such as byte-pair encoding that learns frequent fragments from data, so common words are one token and rare words are split into several.
Why does it matter?
Tokens are the unit that most things are measured in:
- Context window: the model can handle only a limited number of tokens per request (your prompt plus its answer). See What is a Context Window?.
- Cost: hosted model APIs commonly price by tokens processed and generated, often with different rates for input and output. Check your provider's current pricing.
- Speed: longer outputs take longer because tokens are produced one after another.
- Odd behaviour: because models see tokens rather than letters, tasks like counting letters in a word or reversing a string can go wrong.
How does it work?
A rough rule of thumb for English is that one token is about four characters, or about three-quarters of an average word. That is only an estimate: code, numbers, and non-English languages often need more tokens for the same meaning, and every model family has its own tokenizer, so the same text can have a different count in different models.
Example (illustrative — exact splits depend on the tokenizer):
"Tokenization is useful."
→ ["Token", "ization", " is", " useful", "."] (about 5 tokens)For a precise number, use the tokenizer or token-counting facility published by the model provider you actually use.
Example
You paste a 6,000-word report and ask for a summary. If the report is roughly 8,000 tokens, those count against the context window and against your bill (if billed per token) before the model writes a single word of the summary. If the model's limit is smaller than your prompt plus the reply you want, the request fails or the text must be cut or chunked.
Practical consequence: be deliberate about what you include. Sending only the relevant sections is cheaper, faster and often more accurate than sending everything.
When should I use it?
Think in tokens when you are:
- estimating cost for an application that will make many calls;
- deciding how much source material fits in a prompt;
- designing Chunking: How to Split Documents for RAG for What is RAG (Retrieval-Augmented Generation)?, where chunk sizes are commonly set in tokens;
- debugging truncated answers (the output limit was hit).
Common mistakes
- Equating tokens with words. They differ, and the ratio changes by language.
- Forgetting output tokens. The reply counts toward the limit too.
- Reusing one model's token count for another. Tokenizers differ.
- Ignoring system prompts and tool definitions. Hidden instructions also consume tokens.