A/B Testing
Use controlled comparisons while respecting uncertainty and interference. · Data & Experimentation · Lesson 38 · 4 min
A/B Testing · 4 min
Situation
The variant is ahead after one day.
A signup experiment shows a 6% relative lift. Marketing wants to launch immediately. The sample is small, weekday behavior may differ, and paid conversion takes two weeks to observe.
An early lead is a result in progress, not automatically a reliable product improvement.
Mental model
Compare groups created by random assignment.
Define the hypothesis, eligible population, randomization unit, control, variant, primary metric, and guardrails before starting. Decide what difference would be worth acting on and how much evidence you need.
Randomization helps make groups comparable on average. It does not fix broken instrumentation, contamination, or a poorly chosen outcome.
Example
State a testable hypothesis.
“Showing a concrete example during signup will increase the share of eligible new accounts that complete a first useful task within seven days, without increasing support contacts.”
This connects a change to a behavior and a window. Measuring clicks on the example alone would show interaction, not whether the intended value improved.
Design
Choose the unit that fits the interaction.
If colleagues share a workspace, assigning individuals to different collaboration flows can contaminate the experience. Account-level assignment may be more appropriate, though it changes sample-size needs.
Marketplaces and networked products can have interference across participants. Discuss an appropriate experiment design with data specialists rather than assuming user-level randomization always works.
Interpret
Estimate the effect and its uncertainty.
Look at the magnitude, confidence interval or other planned uncertainty measure, and practical importance. A statistically detectable difference can be too small to justify complexity.
Repeatedly checking and stopping when a conventional significance threshold is crossed can inflate false positives unless the analysis method accounts for that stopping rule. Follow the preplanned approach or document a justified change.
Failure case
A winning proxy hides a losing outcome.
A variant may increase signup completion while attracting lower-intent users or making refunds more common. Check guardrails and relevant delayed effects.
Verify assignment balance and exposure logging. An unexpected control-to-variant sample ratio can signal a broken experiment before the outcome metrics deserve interpretation.
PM decision
Choose among launch, iterate, and inconclusive.
A low-traffic product may not have enough evidence to distinguish a worthwhile effect. “No significant result” does not prove no effect, and a positive point estimate does not prove success.
Use the interval, cost, reversibility, and other evidence to decide. If you ship under uncertainty, state that clearly and monitor the actual risk.
Remember this
An experiment answers a defined question under conditions.
It does not certify the feature for every audience and future context. Good experimentation combines sound design, reliable measurement, and a decision threshold that matters to the product.