Chapter 12: Measure Progress and Find Out What's Working

Section 1

Write the question before running the test

A specific question

“Does a clearer product explanation help eligible recipients reach the product page?”

Contrast with: “Which email is better?”

A specific question makes it easier to choose the variable and the measure.

Section 2

Match the measure to the decision

Connect the test to Nick’s seven retention metrics

Choose the customer outcome first, then use engagement to diagnose the journey.

Subscription-to-purchase rate

Does the change turn noncustomer subscribers into first-time buyers? Compare signup cohorts over the same observation period.

30-day purchase rate

Does onboarding lead to a first purchase sooner? Include only noncustomer-at-signup cohorts with a complete 30-day window.

Repeat purchase rate

Do more customers place another order? Use the same customer denominator and follow-up window.

Time between orders

Do customers return sooner? Define first-to-second or successive orders; buyers who have not returned are not represented in completed intervals.

Customer lifetime value

Does observed revenue per customer grow over comparable cohort ages? Keep observed revenue separate from predictions and profit.

Revenue per subscriber

Does each subscriber generate more revenue? State the period, subscriber denominator, revenue definition, and attribution rules.

Returning-customer revenue percentage

Does more revenue come from returning customers? Divide consistently classified returning-customer revenue by total revenue for the same period; acquisition changes can shift the share too.

In the hypothetical examples here, a higher open rate can reflect curiosity without more purchases; privacy-generated opens also obscure intent. A higher popup signup rate can add people who never buy, or who only buy with margin-eroding incentives. Check first-purchase conversion, revenue per subscriber, and contribution before calling either change a commercial win.

A metric can improve while another worsens. Choose a primary measure and a small set of guardrails, such as complaints, unsubscribes, or margin.

Section 3

Isolate what you can

Where possible, randomly assign comparable eligible recipients to test groups. Change one meaningful variable when the purpose is to understand that variable.

Examples of variables

• Headline
• Product explanation
• Offer
• Destination
• Timing

If you change the offer, audience, layout, and timing together, you can evaluate the combined approach, but you cannot confidently attribute the difference to one element.

Section 4

Give the comparison time

Choose the evaluation window in advance and allow for the buying behaviour you are measuring. Avoid declaring a winner from an early fluctuation.

Small audiences and few orders can leave substantial uncertainty. A small difference is not automatically a reliable result.

Section 5

Understand attribution and incrementality

Attribution

Assigns credit according to a platform’s rules.

Incrementality

Asks what happened because of the marketing that would not otherwise have happened.

Published case studies report outcomes, not controlled experiments; they do not by themselves establish incremental lift. Randomized holdouts can help evaluate incremental effects when designed appropriately. Simple before-and-after comparisons can be affected by seasonality, acquisition, stock, promotions, and other changes.

Do not withhold essential transactional or service communication.

Section 6

Keep a learning log

Section 7

Read an illustrative result

Illustrative scenario

A campaign generated more attributed revenue during a major storewide promotion.

Did the email change cause the increase?

Not enough information. The offer, traffic, buying intent, and other channels may have changed. Document those factors and choose a better comparison before assigning credit.