Data Traps
Challenge a plausible conclusion without getting stuck forever. · Data & Experimentation · Lesson 39 · 3 min
Data Traps · 3 min
Situation
Customers using feature X retain better.
The team wants to require feature X during onboarding. But committed customers may be more likely both to discover X and to stay.
Correlation identifies a relationship. It does not establish that forcing the feature will cause retention. Ask what alternative explanation could produce the same pattern.
Selection bias
The observed group may be unusual.
Surveying only active customers misses people who left. Studying support tickets overrepresents users who chose to contact support. Comparing voluntary beta participants with everyone else can confuse motivation with treatment effect.
Name who is absent from the data and whether that absence affects the decision.
Small samples
A dramatic percentage can be one person.
If one of five users buys, conversion is 20%. If two buy, it doubles. That does not make the apparent change meaningless, but it makes the uncertainty large.
Show counts with rates. Avoid treating a tiny segment's movement as a stable effect, especially after searching through many possible segments.
Timing
Seasonality and concurrent changes matter.
A shopping feature launched before a holiday may coincide with a predictable demand rise. A weekday-to-weekend comparison can distort usage for a work product.
Compare appropriate periods and check other changes. A year-over-year comparison can help with seasonality but may still contain product, audience, and market differences.
Incentives
People adapt to the metric.
If support is rewarded only for closing tickets quickly, agents may close unresolved cases. The metric improves while the customer experience degrades.
Track reopened cases and resolution quality, and inspect actual conversations. A measurement system can change the behavior it claims merely to observe.
Remember this
Ask what else could explain the result.
Check selection, sample size, timing, causality, and incentives. Then choose a proportional next step: more analysis, a small experiment, or a reversible decision with explicit uncertainty.