What is it?
A language model doesn't write words directly; at each step it produces a probability for every possible next token. Decoding is how one token is picked. Pick the likeliest every time and output is steady but can be repetitive. Add randomness and it becomes more varied — and more error-prone.
Temperature and top-p are the two dials you will meet most.
Also known as: temperature, top-p, top p sampling, nucleus sampling, top-k sampling, greedy decoding, beam search, decoding strategies.
Technical definition
Common methods: greedy (argmax); temperature scaling (logits divided by T before softmax); top-k (keep k highest-probability tokens); nucleus/top-p (keep the smallest set whose cumulative probability ≥ p; Holtzman et al., 2019); min-p; typical sampling; beam search (keep B best partial sequences); repetition/frequency/presence penalties; logit bias. Temperature 0 is often treated as near-greedy but identical output is not guaranteed across hardware, batching or provider changes. Constraints on allowed tokens are covered in Constrained Decoding (Guided Generation); aggregation across samples in Self-Consistency (Sampling and Voting).
Entity type: concept · Freshness class: evolving.
When should I use it?
Tune these whenever output variability matters: lower for repeatable tasks, higher for ideation.
Implementation patterns
- Low temperature for extraction, classification and code.
- Moderate temperature plus top-p for drafting and brainstorming.
- Change one of temperature or top-p at a time, as providers often advise.
- Sampling for diversity, then rank or vote (Best-of-N Sampling).
Technologies and frameworks
Commonly associated: provider API parameters, Hugging Face generate(), llama.cpp sampler settings, vLLM SamplingParams. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Start with provider defaults.
- For factual/extraction tasks, lower temperature.
- Change one parameter at a time and compare on a fixed test set.
- Add schema constraints for structure.
- Record settings with each evaluation run.
Common failure modes
- Assuming temperature 0 guarantees identical outputs.
- Cranking temperature to "reduce hallucination" — it does not; low temperature does not remove factual errors either.
- Repetition loops with greedy or very low temperature.
- Setting both temperature and top-p aggressively and getting odd text.
Troubleshooting
- Output repeats phrases — Raise temperature slightly or add a repetition/frequency penalty if available.
- Output is erratic or off-topic — Lower temperature or top-p.
- Same prompt gives different JSON — Use structured outputs / constrained decoding, not just low temperature.
Contrasted with
- Constrained Decoding (Guided Generation) — Sampling controls randomness; constrained decoding controls which tokens are allowed.
Questions this page answers
- What is temperature in an LLM?
- What is top-p sampling?
- What is the difference between temperature and top-p?
- Does temperature 0 make output deterministic?
- How do I stop my LLM repeating itself?
Typical goals: set generation parameters; make output more deterministic; make output more creative; debug repetitive output.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Holtzman et al., The Curious Case of Neural Text Degeneration (2019) — arXiv:1904.09751
- Nguyen et al., Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs (2024) — arXiv:2407.01082