AightBits

A Clear, Practical Introduction to AI Sampling Techniques

Audience

• Prompt Engineers
• LLM Application Developers
• AI Content Designers / UX Writers for LLMs
• Product Managers & Technical PMs in AI Tools
• AI-Powered Tool Builders / No-Code or Low-Code Integrators
• LLM QA Specialists / Content Reviewers
• Educators and Technical Writers Explaining LLM Concepts

Introduction

Sampler settings control how an AI model selects words when generating text.
They don’t change what the model knows—they influence how the model chooses among possible next words based on their likelihood.

By tuning these settings, you can encourage the model to produce:

• More predictable, focused writing, or
• More varied, creative writing, sometimes at the cost of factual accuracy or coherence.

This guide introduces four common settings—Temperature, Top-k, Top-p, and Minimum Probability Filtering—without deep technical details.

Adjusting How Likely Different Outputs Are: Temperature

Temperature controls the randomness in the model’s selection of next words:

• Lower temperatures (<1.0) make high-probability words even more likely to be chosen, resulting in more deterministic outputs.
• Higher temperatures (>1.0) flatten the probability distribution, making lower-probability words more likely to be selected.
• At temperature = 1.0, the model samples words directly according to their original probabilities without adjustment.

Behavior:

• Lower temperature → more focused, predictable output.
• Higher temperature → more varied and unexpected output.

Warning:
Higher temperatures increase the chance of incoherent or broken text, especially above about 1.5. Exact effects vary depending on the model and prompt.

Typical recommended range: 0.2 – 1.0

Limiting How Many Options Are Considered: Top-k and Top-p

After temperature adjustment, the model can limit its possible next-word choices using either Top-k or Top-p settings (usually only one is used at a time, as they can be redundant when combined).

Top-k Sampling

The model ranks all possible next words by likelihood and selects only the top k words. The probabilities of these selected words are adjusted (normalized), preserving their relative likelihood, and the model then randomly picks from them.

Behavior:

• Small k (e.g., 10–20) → more focused, consistent outputs (but very small k values may produce repetitive results).
• Larger k (e.g., 50–100) → more variety and creativity.

Note:
One limitation of Top-k is that it uses a fixed cutoff regardless of the context, which may sometimes exclude contextually important but lower-probability words.

Typical recommended range: 20 – 100

Top-p Sampling (Nucleus Sampling)

Instead of picking a fixed number of words, Top-p includes enough top-ranked words to cover a certain cumulative probability (confidence threshold) (e.g., 90%). Probabilities are adjusted, and the model randomly selects from this group, favoring higher-probability words.

Behavior:

• Lower p (e.g., 0.7) → tighter, safer outputs.
• Higher p (e.g., 0.95) → broader, more varied choices.

Tip:
Top-p automatically adapts the number of tokens considered based on the probability distribution—fewer tokens when the model is confident about a few options, more tokens when probability is distributed across many options. This dynamic “vocabulary size” is why Top-p is often preferred over Top-k.

Typical recommended range: 0.8 – 0.95

Filtering Out Extremely Unlikely Options: Minimum Probability Filtering

Minimum Probability Filtering (sometimes called “min_p”) removes extremely unlikely or nonsensical words (such as rare misspellings or obscure terms) from consideration entirely, helping to avoid irrelevant or strange outputs.

Important:
Minimum Probability Filtering is typically an advanced setting found in customized or complex systems. Beginners rarely encounter or need to adjust this, and it is often unavailable in mainstream platforms.

Behavior:

• Very low thresholds (like 0.00001) → almost no visible effect.
• Higher thresholds (like 0.001) → stricter filtering, sometimes causing incomplete sentences or odd phrasing.

Caution:
Setting the minimum too high can cause the model to get “stuck” or make unnatural choices.

Typical active range (if used): 0.00001 – 0.001

How These Settings Work Together

Usual Process

• Temperature adjusts the overall probability distribution.
• Either Top-k or Top-p (typically not both) further limits the words considered.
• Minimum Probability Filtering removes extremely rare or nonsensical options (rarely adjusted by beginners).

For beginners, it’s usually best to start by experimenting with temperature alone, and gradually introduce Top-k or Top-p afterward.

Example Settings

Focused, Reliable Output:
Temperature: 0.3
Top-k: 20

Creative, Surprising Output:
Temperature: 0.9
Top-p: 0.95

Quick Reference Table

Setting	What It Does
Temperature	Adjusts randomness and focus in selecting next words
Top-k	Limits choices to the k most likely words (renormalized probabilities)
Top-p	Limits choices based on cumulative confidence, with dynamic vocabulary size
Minimum Probability Filtering	Removes extremely unlikely or nonsensical word options