Retention and Cohorts
Compare like with like to understand who keeps receiving value. · Data & Experimentation · Lesson 35 · 4 min
Retention and Cohorts · 4 min
Situation
Active users rise while new users struggle.
A product grows from 20,000 to 25,000 monthly active users after a large acquisition campaign. The team celebrates. Yet fewer new users return after their first week.
Acquisition can lift the total while retention worsens. A growing average does not reveal the experience of each starting group.
Mental model
A cohort shares a starting condition.
Group users by signup week, first purchase month, or another meaningful starting event. Then measure whether they return to a defined valuable behavior after a defined interval.
Specify the return event and unit. A passive page load is different from completing useful work, and user retention is different from account or revenue retention.
Example
Compare cohorts at the same age.
| Signup cohort | Started | Returned in week 4 |
|---|---|---|
| June A | 1,000 | 300 (30%) |
| July B | 2,000 | 400 (20%) |
These are illustrative figures. July produces more returning people but a lower return rate. Both facts matter. Comparing July's week 1 with June's week 4 would answer a different question.
Define the window
“Thirty-day retention” is ambiguous.
It can mean activity on exactly day 30, activity during a day-30 window, or activity on or after a threshold. These definitions yield different numbers.
Choose a cadence that fits the job. A monthly payroll product should not be judged by daily use. State the definition next to the result so others can reproduce it.
Investigate
A cohort change can have several causes.
Did acquisition bring a less suitable audience? Did onboarding change? Did tracking break? Are newer cohorts fully old enough to observe the target interval?
Segment by relevant characteristics and compare equivalent age windows. A cohort whose observation period is incomplete should not be treated as having failed to return.
Failure case
Retention looks good because the denominator changed.
Removing inactive users from the starting cohort makes retention look artificially strong. Defining return as any automated event can also inflate it.
Keep cohort membership stable under the chosen definition. If a data correction changes the denominator, disclose the change instead of presenting it as product improvement.
PM decision
Diagnose the kind of value loss.
If users complete the first task and never return, investigate whether the need recurs. If the need recurs but they choose another tool, investigate value, reliability, and switching behavior.
A reminder campaign may help forgotten useful behavior. It will not repair a workflow that disappointed users the first time.
Remember this
Retention combines people, behavior, and time.
Compare cohorts at the same age using a return event that reflects the product's job. Read acquisition volume and retention quality together.