What an LLM Actually Does
Understand why useful generation is not the same as verified knowledge. · AI Product Management · Lesson 41 · 3 min
What an LLM Actually Does · 3 min
Situation
The answer sounds confident and contains a made-up detail.
A ChatGPT-style assistant writes a convincing explanation of a company's refund policy. The wording is fluent, but the refund window is wrong.
The experience can feel like consulting a knowledgeable person. The underlying generation process does not guarantee that every statement has been checked against an authoritative source.
Mental model
Generate a continuation from learned patterns and context.
A language model learns patterns during training and generates output conditioned on the input and available context. For text, it repeatedly selects tokens, which can be words or pieces of words.
Training and post-training shape useful behaviors, including following instructions and solving tasks. The mechanism can support sophisticated reasoning while still producing factual errors.
Example
The model needs access to current facts.
If a customer asks for today's order status, a model's general training cannot establish the current state of that order. The product needs an authorized data source or tool.
The model can help interpret the question and explain the returned status. The database or service remains the authority for whether the order shipped.
Failure case
Familiar wording substitutes for evidence.
A policy question resembles many examples the model has encountered. It may generate a plausible policy even when your actual policy is absent from context.
Ask whether the product should retrieve evidence, request clarification, or decline to answer. The right behavior depends on what is missing and the consequence of being wrong.
PM decision
Define the role of generation.
Drafting a friendly explanation from verified facts is different from inventing the facts. Separate those responsibilities in the product design and evaluation.
For a refund assistant, evaluate whether it follows the supplied policy, identifies missing details, and avoids claiming actions occurred without confirmation. Fluency alone is the wrong quality bar.
Remember this
A convincing answer is an output, not proof.
Use models for the capabilities they demonstrate on your task. Add reliable data, constrained actions, and evaluation where the product requires correctness.