Embeddings and Vector Search
Use semantic similarity without confusing it with exactness or truth. · AI Product Management · Lesson 48 · 3 min
Embeddings and Vector Search · 3 min
Situation
The user says “change my plan”; the article says “subscription.”
A keyword search may miss a useful help article because the wording differs. A semantic search can find related meaning even when the exact terms do not match.
That can improve discovery, but some tasks need exact identifiers and strict filters more than conceptual similarity.
Mental model
Represent meaning as a location for comparison.
An embedding converts content into a numerical representation. Similarity search compares these representations to find items that are close under the chosen model and distance measure.
You do not need vector mathematics to ask whether the returned material is useful. Similarity is a retrieval signal, not a judgment that a document is correct or authorized.
Example
Match the method to the query.
“Help me stop paying every month” may benefit from semantic matching to cancellation guidance. An order ID such as ORD-842 needs exact lookup. “Policy for France” needs a region constraint.
A hybrid approach can combine keyword and semantic signals with metadata filters. Evaluate whether the combination improves the actual search task.
Failure case
Similar but inapplicable is still wrong.
A policy for a neighboring country can sound almost identical while differing in the rule that matters. An outdated article may rank well because its wording matches the question.
Use authoritative metadata, access filtering, and version handling. Do not expect semantic similarity to enforce product or policy constraints by itself.
PM question
What is the retrieval quality bar?
Build examples with expected useful results, exact-ID queries, ambiguous phrasing, and no-answer cases. Evaluate whether relevant material appears near the top and whether irrelevant material could mislead a downstream answer.
Inspect failures by query type. A better embedding model may not fix poor documents, missing metadata, or an inappropriate chunking strategy.
Remember this
Semantic search finds related content.
Combine it with exact constraints where the task requires them. The PM's job is to define relevance for the user's decision, not to maximize an abstract similarity score.