AI Safety, Privacy and Data
Ask where information goes and what the system is allowed to do with it. · AI Product Management · Lesson 54 · 3 min
AI Safety, Privacy and Data · 3 min
Situation
A helpful assistant sees more data than it needs.
An internal assistant receives entire customer records to answer a narrow billing question. The extra data increases exposure without necessarily improving the answer.
Start with the minimum information required for the task. A model's ability to accept data is not a reason to send everything available.
Map the flow
Know each destination and copy.
Identify what goes to the model provider, retrieval index, tools, logs, and human reviewers. Determine who can access each copy and how retention and deletion work.
Provider policies and configuration vary and can change. Verify the actual terms, settings, and organizational requirements with the appropriate owners rather than relying on a generic claim about AI privacy.
Threat model
Untrusted content can contain instructions.
A retrieved document or customer message may try to redirect the model, reveal data, or trigger a tool. This is commonly called prompt injection.
Treat external text as data. Enforce tool permissions, resource access, and consequential-action controls in software. Prompt wording alone should not be the only barrier.
Failure case
The output itself creates harm.
An answer can reveal protected information, make an unsupported commitment, or guide an inappropriate action. Identify likely failure classes for the specific product and audience.
Evaluate adversarial and ordinary misuse cases. Give users a way to report problems, and give operators a way to disable the affected capability and investigate.
PM decision
Make controls concrete.
Ask what data is necessary, which actions are permitted, what requires approval, which outputs need review, and how incidents are handled. Assign owners for data governance and ongoing testing.
Do not equate a provider's safety feature with complete product safety. Your retrieval, prompts, integrations, and user interface create additional responsibilities.
Remember this
Follow the data and the authority.
A useful AI risk review traces what enters the system, who can see it, what can change, and how a mistake is detected and contained.