How LLMs Generate Text, One Token at a Time
Tokens, probabilities, sampling and context windows — the mental model every GenAI engineer needs.
Generative AIBeginner2 min readUpdated 11 Oct 2026
The whole trick: predict the next token
A large language model is trained to answer one question over and over: given everything so far, what token comes next? Generating a paragraph is just repeating that step.
next-token prediction
Data pipelines must be idempotent so that a rerun never doubles revenue
Done — the model emitted an end token.
Tokens, not words
Text is split into tokens — common words are one token, rare words become several pieces. Prices, context windows and output limits are all measured in tokens.
"Snowflake COPY INTO is idempotent"
→ ["Snow", "flake", " COPY", " INTO", " is", " idem", "potent"]From scores to a choice
- 1
Score every candidate
The model outputs a score (logit) for every token in its vocabulary. - 2
Turn scores into probabilities
Softmax converts scores into a probability distribution. - 3
Sample one token
Temperature and top-p decide how adventurous the choice is. - 4
Append and repeat
The chosen token joins the input, and the model predicts again — until a stop token or a length limit.
| Setting | Low value | High value |
|---|---|---|
| Temperature | Focused, repeatable — good for SQL, extraction | Varied — good for brainstorming |
| Top-p | Only the most likely tokens | Wider vocabulary |
| Max tokens | Short, cheap answers | Long answers, higher cost |
Finished this note?