From Data to Recommendation
Move from an observation to a bounded product decision. · Data & Experimentation · Lesson 40 · 3 min
From Data to Recommendation · 3 min
Situation
More activity, less repeat use.
Imagine a Spotify-style listening product testing a more aggressive autoplay flow. Listening minutes rise 8%, while 30-day retention falls four percentage points among fully observed new-user cohorts.
These numbers are hypothetical. The point is to practice a decision where the obvious engagement win conflicts with a longer-term outcome.
Verify
Make the comparison trustworthy.
Confirm cohort age, return-event definition, exposure, and instrumentation. Are minutes measured consistently? Is the retention change four percentage points or 4% relative? Are the compared users randomized or merely before-and-after groups?
Check whether the apparent decline is larger than expected uncertainty. A conflicting result deserves investigation before either celebration or rollback by reflex.
Investigate
Separate the plausible mechanisms.
Segment by new and established listeners, device, and relevant listening context. Examine whether additional minutes are intentional or unattended autoplay.
Pair the numbers with feedback and behavioral evidence such as skips, stops, and autoplay disabling. More consumption can reflect better recommendations, reduced control, or a measurement change.
Interpret
Explain what the evidence supports.
Suppose the decline concentrates among new mobile users, autoplay opt-outs rise, and interviews describe difficulty stopping or choosing the next track. That pattern supports a control-and-expectation problem.
It still does not prove every extra minute is unwanted. State the narrower interpretation and the evidence that would challenge it.
Recommend
A decision with a next test.
“Pause expansion to new mobile users and test a clearer autoplay choice. Keep the current experience for unexposed users while we compare retention, intentional starts, and opt-outs.”
Set the owner, observation window, and decision criteria before the test. If user harm is already clear, act to reduce it while the longer-term measurement matures.
Remember this
The recommendation should be narrower than the uncertainty.
Observation → verification → investigation → interpretation → decision. Explain what you know, what you infer, and why the next action is proportionate to both benefit and risk.