# Tokens, Context Windows and Sampling

> What a token is, why the context window limits everything an LLM sees, and how temperature and top-p change its answers.

- Source: https://akshaythakare.com/notes/generative-ai/tokens-and-context-windows
- Topic: Generative AI
- Level: beginner
- Reading time: 2 min
- Tags: genai, llm, tokens, fundamentals, interview
- Updated: 2026-10-11

## Tokens

Models read and write **tokens**: pieces of words produced by a tokenizer. In English text one token is roughly three-quarters of a word, but code, numbers and other languages often use more tokens per word.

Tokens matter because:
- **cost** is usually priced per input and output token
- **latency** grows with the number of output tokens
- **limits** are counted in tokens

## The context window

The context window is the maximum number of tokens a model can consider in one request: system prompt, conversation, retrieved documents, tool results **and** its own answer.

```text
[system prompt][history][retrieved docs][user question] → [answer]
|------------------- must fit in the context window -------------------|
```

Larger windows help, but more context is not free. It costs more, runs slower, and relevant details can get lost in long inputs. Retrieve and include only what the task needs.

## Sampling: temperature and top-p

At each step the model produces a probability for every possible next token.

| Setting | Effect | Use |
| --- | --- | --- |
| Low temperature (≈0) | Picks the most likely tokens, so output is stable | Extraction, classification, code |
| Higher temperature | More varied choices | Brainstorming, creative writing |
| top-p | Samples only from the smallest set of tokens covering probability p | Limits unlikely choices |

> **Note: Deterministic is not guaranteed**
>
> Even at temperature 0, outputs can differ slightly between calls. Tests should check properties of the answer, not exact strings.