Tokens, Context and Sampling
Connect model settings to memory, variation, cost, and user expectations. · AI Product Management · Lesson 42 · 3 min
Tokens, Context and Sampling · 3 min
Situation
The assistant forgets an earlier requirement.
A long conversation contains a customer's constraints, several documents, and many tool results. Later, the answer ignores a detail near the beginning.
The product may have truncated or summarized context, or the model may not have used the available detail reliably. A large context window is not equivalent to perfect memory.
Mental model
Context is the material available for this response.
Tokens are units the model processes; a word may use one or several tokens. The context window limits the material the model can work with, subject to model-specific accounting for inputs, outputs, and other tokens.
Instructions, conversation, retrieved passages, and tool results compete for that space. Cost and latency depend partly on how much work the model processes and generates.
Example
Choose relevant context rather than everything.
For a support answer, include the applicable policy, the relevant account facts, and the user's question. Dumping an entire help center into context can add cost and distracting contradictions.
If a long thread is summarized, preserve decisions and critical constraints explicitly. Test whether the compression loses details needed for later actions.
Sampling
Variation is a product choice with limits.
On models that expose temperature or related sampling controls, settings can change output variability. More variation can help brainstorm alternatives; less variation may suit repeated extraction tasks.
Lower randomness does not guarantee factual correctness or identical output in every system. Some models do not support the same controls. Verify actual capabilities rather than treating one parameter as universal.
Failure case
Hiding a context limit from the user.
If an upload is too large, silently discarding pages creates an answer that appears more comprehensive than it is. Show the supported scope or process the document in a defined way.
Ask how the system reports incomplete input, exhausted output limits, and missing conversation history. These are UX states, not only API errors.
Remember this
Manage context as a product resource.
Keep critical information explicit, evaluate long and short cases, and match variability to the task. Neither a larger window nor a lower temperature replaces verification.