Instrumentation
Make sure the event means what the decision assumes it means. · Data & Experimentation · Lesson 36 · 3 min
Instrumentation · 3 min
Situation
The new flow doubles “orders completed.”
The dashboard looks excellent, but finance reports no change in real orders. An event was added to the confirmation page and fires again on refresh.
Tracking errors can produce convincing product stories. Instrumentation is part of the product's decision system and deserves explicit requirements.
Mental model
An event is a recorded occurrence.
An event name identifies what happened. Properties describe relevant context. User or account IDs connect events to an entity. Timestamps place them in time.
Define whether time means the action time or the time received by analytics. Offline clients and delayed processing can make those different.
Example
Specify a business event precisely.
For order_completed, define whether it means payment accepted, order created, or fulfillment finished. Include stable order ID, relevant product context, and an appropriate identity key.
Prefer authoritative server confirmation for outcomes the server owns. Client events remain useful for interface behavior, but they should not silently redefine a business transaction.
Failure case
Missing identity splits one journey.
A visitor browses anonymously, signs in, and purchases. If the system cannot connect the relevant events under its permitted identity rules, the funnel may show separate people.
The opposite mistake merges different people. Document identity handling and privacy constraints with data and engineering teams rather than making assumptions in the analysis.
PM decision
Test the tracking plan before the experiment.
Walk through success, failure, refresh, retry, and duplicate-action cases. Verify event properties, timestamps, deduplication, and counts against a known source.
Collect only what the analysis needs. Avoid adding sensitive free text or personal data simply because a property field is available. Assign an owner for schema changes.
Remember this
A number is only as clear as its event definition.
Write instrumentation acceptance criteria alongside product behavior. If the event cannot reliably distinguish success from an attempted action, do not use it as proof of success.