Top-p
Why You Need to Understand Top-p
When a model generates each word, it's actually "sampling" from all possible words by probability.
The problem: the probability distribution is long — the "most likely word" may account for 80%, "second most likely" for 10%, and everything else adds up to only 10%. If you sample from all words every time, occasionally you'll pick low-probability tail words, causing output to deviate.
Top-p solves this: controlling the range of the "candidate pool" rather than adjusting the shape of the probability distribution.
What Is Top-p
One-line definition: Top-p limits the sampling candidate range, selecting only from the smallest set of words whose cumulative probability reaches the threshold.
When top-p = 0.9:
- The model first lists all candidate words, sorted by probability
- Starting from the highest, accumulate until cumulative probability reaches 90%
- Only select the next word from the words covered by that 90% probability
- The remaining 10% of low-probability tail words are directly excluded
Analogy: Like a teacher grading an exam, only grading the top 90% of the most carefully written parts of the answer sheet — directly ignoring the most illegible parts.
Top-p vs Temperature
Both control randomness but in different directions:
| Parameter | Controls What | Analogy |
|---|---|---|
| Temperature | Adjusts probability distribution smoothness | Adjusting "risk tolerance" |
| Top-p | Limits the candidate word range | Adjusting "candidate pool width" |
Simply put:
- Temperature = "dare to pick low-probability words"
- Top-p = "how wide is the pool of words allowed to be selected"
About Token consumption: these two parameters mainly affect output diversity and stability, not directly affecting Token count, but can indirectly affect Token consumption through output length changes.
How to Do It: When to Adjust Top-p
Suitable for lowering Top-p (0.7-0.9):
- Wanting some variation but not too far off
- Being sensitive to low-probability absurd words
- Needing to control divergence boundaries
Suitable for keeping default Top-p (usually 1.0 or unset):
- Already adjusting Temperature
- Wanting the model to have more autonomous space
Practical advice:
- If already adjusting temperature, usually keep top-p at default first
- Generally not recommended to significantly increase both simultaneously, otherwise output will noticeably destabilize
- In many products, temperature is sufficient, Top-p is more for fine-tuning
Remember this: Top-p is the control valve for "candidate pool width" — it controls "which words are eligible to be selected," not "how conservative the selected words are."
Related terms: Temperature
Related Products