RAG
Use retrieval to supply evidence, then evaluate both finding and answering. · AI Product Management · Lesson 47 · 4 min
RAG · 4 min
Situation
The assistant needs your current policy.
A support assistant knows general customer-service language but not the latest internal return rules. Putting the policy in the prompt for every request may be impractical as the library grows.
Retrieval-augmented generation, or RAG, retrieves relevant information and provides it as context for an answer. It changes what evidence the model can see.
Process
Retrieve → provide context → generate.
A query is used to find relevant passages or records. The system selects and supplies them with the question and instructions. The model generates an answer using that material.
Retrieval can use keywords, semantic search, filters, or a combination. RAG does not necessarily require a vector database, and it does not retrain the model on each document.
Example
A policy answer needs the right version.
A customer asks whether a discounted item can be returned. The system retrieves the applicable policy for the region, product, and purchase date, then provides that passage to the model.
The answer should explain the rule and link to supporting evidence where useful. If exceptions depend on missing purchase details, asking a question may be better than guessing.
Failure point
Retrieval can fail before generation starts.
The relevant document might be absent, outdated, split into unhelpful chunks, or ranked below a similar but inapplicable policy. The model cannot reliably quote evidence it never received.
Test whether the correct supporting passages are retrieved for representative questions. Inspect no-result cases and conflicts, not only polished final answers.
Failure point
Good retrieval does not guarantee a good answer.
The model can misread a passage, combine incompatible rules, or add an unsupported claim. A citation can point to a real page that does not support the sentence.
Evaluate groundedness: does the supplied evidence actually justify the answer? Also evaluate whether the answer is complete enough to help with the user's task.
Product boundary
Permission filtering comes before exposure.
A search index containing internal documents must honor the user's access. Do not retrieve protected material into context and rely on the model to keep it secret.
Treat retrieved text as evidence, not instructions with authority over the application. A malicious instruction inside a document should not change tool permissions or data access.
PM decision
Separate the two quality questions.
Can the system find the right evidence? Can it answer correctly from that evidence? Use separate diagnostics so the team does not keep rewriting prompts to repair a retrieval problem.
Define refresh and deletion behavior, source ownership, no-answer UX, and a way to report incorrect answers. A knowledge base is an operating responsibility.
Remember this
RAG supplies context, not guaranteed truth.
It can improve access to current or private knowledge. It cannot by itself fix bad source material, missing permissions, faulty retrieval, or unsupported generation.