# How LLMs Generate Text, One Token at a Time

> Tokens, probabilities, sampling and context windows — the mental model every GenAI engineer needs.

- Source: https://akshaythakare.com/notes/generative-ai/how-llms-generate-text
- Topic: Generative AI
- Level: beginner
- Reading time: 2 min
- Tags: generative-ai, llms, tokens, sampling
- Updated: 2026-10-11

## The whole trick: predict the next token

A large language model is trained to answer one question over and over: *given everything so far, what token comes next?* Generating a paragraph is just repeating that step.

*(Interactive illustration “Token Stream”: see the web page.)*

## Tokens, not words

Text is split into **tokens** — common words are one token, rare words become several pieces. Prices, context windows and output limits are all measured in tokens.

```text
"Snowflake COPY INTO is idempotent"
→ ["Snow", "flake", " COPY", " INTO", " is", " idem", "potent"]
```

## From scores to a choice

  
### Score every candidate
The model outputs a score (logit) for every token in its vocabulary.
  
### Turn scores into probabilities
Softmax converts scores into a probability distribution.
  
### Sample one token
Temperature and top-p decide how adventurous the choice is.
  
### Append and repeat
The chosen token joins the input, and the model predicts again — until a stop token or a length limit.

| Setting | Low value | High value |
| --- | --- | --- |
| Temperature | Focused, repeatable — good for SQL, extraction | Varied — good for brainstorming |
| Top-p | Only the most likely tokens | Wider vocabulary |
| Max tokens | Short, cheap answers | Long answers, higher cost |

> **Note: Why this matters for engineering**
>
> The model has no database of facts — it predicts plausible text. That is why grounding with retrieval (RAG), structured outputs and evaluations are essential in production systems.