Segmentation
Check whether a changing mix is telling a false story. · Data & Experimentation · Lesson 37 · 3 min
Segmentation · 3 min
Situation
The total falls even though both groups improve.
A product's overall conversion declines. The team assumes the new flow is worse. But mobile and desktop each convert slightly better than before.
The contradiction can be explained by a shift toward the lower-converting group. Aggregate rates combine performance and population mix.
Example
Work through the numbers.
Before: desktop 90/100 = 90%
mobile 10/100 = 10%
total 100/200 = 50%
After: desktop 19/20 = 95%
mobile 36/180 = 20%
total 55/200 = 27.5%In this deliberately exaggerated example, mobile becomes most of the traffic. Both segment rates improve while the total rate falls. The totals are correct; the initial causal interpretation is not.
Mental model
Ask who is inside the average.
Useful segments may include new versus returning, free versus paid, country, platform, account size, and acquisition source.
Choose segments related to a plausible mechanism. Segmenting by everything until something looks exciting creates false discoveries and can leave groups too small to interpret.
Investigate
Separate mix from within-group change.
Compare each segment's volume and rate, and consider what the total would look like under a stable mix. That helps identify whether performance, audience composition, or both changed.
Also check segment definitions. A pricing change may move customers between free and paid groups, making a naive before-and-after comparison hard to interpret.
Failure case
Explaining away the business result.
A mix shift may be the explanation, but the lower overall conversion still affects the business. If a campaign brought poorly matched traffic, acquisition choices need attention.
Do not use segment improvements to dismiss the total. Explain both the product experience within groups and the consequences of how the audience changed.
Remember this
An average is weighted by who shows up.
When a headline metric moves, inspect the relevant segment rates and their sizes. The right recommendation may concern acquisition, product behavior, or both.