Tokens, Context Windows and Sampling
What a token is, why the context window limits everything an LLM sees, and how temperature and top-p change its answers.
Tokens
Models read and write tokens: pieces of words produced by a tokenizer. In English text one token is roughly three-quarters of a word, but code, numbers and other languages often use more tokens per word.
Tokens matter because:
- cost is usually priced per input and output token
- latency grows with the number of output tokens
- limits are counted in tokens
The context window
The context window is the maximum number of tokens a model can consider in one request: system prompt, conversation, retrieved documents, tool results and its own answer.
[system prompt][history][retrieved docs][user question] → [answer]
|------------------- must fit in the context window -------------------|Larger windows help, but more context is not free. It costs more, runs slower, and relevant details can get lost in long inputs. Retrieve and include only what the task needs.
Sampling: temperature and top-p
At each step the model produces a probability for every possible next token.
| Setting | Effect | Use |
|---|---|---|
| Low temperature (≈0) | Picks the most likely tokens, so output is stable | Extraction, classification, code |
| Higher temperature | More varied choices | Brainstorming, creative writing |
| top-p | Samples only from the smallest set of tokens covering probability p | Limits unlikely choices |
Finished this note?