A/B testing in newsletters

In an A/B test, two randomly formed groups from the mailing list receive two versions of the same mailing that differ in exactly one respect. Afterwards you compare which version performed better. That sounds simple – but most mistakes happen not when sending but when interpreting the figures.

What can be tested

Subject line or preview text – better measured by clicks than by opens (see below),
sender name, for example the club’s name versus the name of the chair,
structure: one main topic at the top versus an overview of several topics,
call to action: the wording or placement of a button,
sending time: day of the week or time of day.

Five rules for a fair test

Only one change. If subject line and image both differ, you will not know afterwards what made the difference.
Random groups. Not “the first 500 on the list” versus “the next 500” – lists are often sorted by sign-up date or alphabetically. Most sending platforms split randomly.
Same time. Send both versions at the same time, unless the timing itself is what is being tested.
Decide on the metric beforehand. Decide before sending what you will measure by – and stick to it.
Enough recipients and enough time. Small groups deliver chance results; anyone who evaluates after an hour mainly measures the fast readers.

The right metric

Many platforms automatically evaluate subject line tests by open rate. That is unreliable today: with “Protect Mail Activity”, Apple Mail loads remote content privately in the background as soon as an email arrives – this looks like an open without anyone having read anything. Where possible, therefore, measure by clicks or by what happens afterwards (sign-ups, enquiries, purchases). More on this: Open rate and click rate.

Is the difference more than chance?

Even two identical versions never achieve exactly the same click rate. Whether a difference goes beyond chance is checked with a test for the equality of two proportions. The handbook of statistical methods of the US National Institute of Standards and Technology (NIST) describes it for sufficiently large samples:

Calculate the pooled proportion: p = (clicks A + clicks B) ÷ (recipients A + recipients B).
Calculate the standard error: SE = √( p × (1 − p) × (1/recipients A + 1/recipients B) ).
Calculate the test statistic: z = (click rate B − click rate A) ÷ SE.
Compare: if |z| is greater than 1.96, the difference is statistically significant in a two-sided test at the usual level of 5% (value 1.960 from the table of the standard normal distribution in the same handbook).

Example 1 (large groups): version A goes to 2,000 recipients, 100 click (5.0%). Version B goes to 2,000 recipients, 130 click (6.5%).

p = 230 ÷ 4,000 = 0.0575
SE = √(0.0575 × 0.9425 × (1/2,000 + 1/2,000)) ≈ 0.00736
z = (0.065 − 0.050) ÷ 0.00736 ≈ 2.04 → greater than 1.96: the difference is significant.

Example 2 (small groups): 500 recipients each, 25 versus 32 clicks (5.0% versus 6.4%).

p = 57 ÷ 1,000 = 0.057
SE = √(0.057 × 0.943 × (1/500 + 1/500)) ≈ 0.0147
z = (0.064 − 0.050) ÷ 0.0147 ≈ 0.95 → less than 1.96: the difference may well be chance.

Almost the same gap in percentage points – but a reliable result only with the large groups. For very small groups, the NIST handbook names an exact test (Fisher’s), which statistics programs calculate.

What small mailing lists can do

Anyone with only a few hundred recipients will rarely reach reliable results with single tests. Three ways out:

Repeat instead of testing once. Check the same question (such as “main topic at the top or overview?”) alternately over several issues and add up the results.
Test big differences. Two fundamentally different approaches are more likely to show a measurable difference than two almost identical subject lines.
Ask instead of measure. A short question in the newsletter (“What would you like to read more often?”) sometimes yields more than any statistic.

Common mistakes

Testing and immediately sending the “winner” to the rest, even though the difference is not significant.
Many variants at once – the more comparisons, the more likely one of them is “significant” by chance.
Generalising results: what worked in an issue in December need not work again in June.
Not writing the test down. Record question, versions, group sizes, metric and result; otherwise you will test the same thing again in a year.