Split testing emails explained

Why split test email campaigns?

You can test your email ideas and get real data on what is most effective. 

How to split test emails?

The first step is to decide which variable you want to test. Examples of variables include:

Subject line - try different wording, questions or phrases to see what gets the most attention. 

Sender name – see if your recipients respond better to an email from a specific person or a generic company email address.

Content – try different versions of the email content and find out what generates higher click-throughs or responses. 

Send time – identify when your recipients are most likely to open your emails.

The most common variable to test is the subject line. Since the email subject is usually the first thing that people see in their inbox, this is a crucial aspect to get right. The email subject needs to grab their attention and compel them to open the email.

This week we ran a split test on the subject line for an email we sent on behalf of a client. The results were quite illuminating. Our first subject line choice was fairly standard so we thought we’d also test something a little less ‘safe’. We sent the two versions of the email equally split to 25% of the total database. Interestingly, our second email choice resulted in 86% more opens than the original. Based on this insight we then sent this version to the remaining 75% of people on the database.

Once you’ve decided on your variable and drafted your two email campaigns, you’ll need to determine how you’re going to measure the test. Are you testing for opens or click-throughs? The type of variable you are testing will probably inform your measure. For example, if you’re testing subject lines, then the best test will be open rates whereas if you’re altering content, then a better measure will be click-throughs. 

Next, choose whether to send your two campaigns to the whole list or a segment. If your database is quite large or this is a one-off campaign we recommend testing a portion, say 25% of the total.

Finally, if your email marketing client allows it, decide how long you want to run the test for. You need to allow sufficient time for your recipients to have the opportunity to open the email.

When the results are in you can send the winning email to the remainder of your database or use the data to prepare future campaigns.

Best practice tips for split testing emails

Ensure that you only change one variable. You can test variables in future A/B tests.

Unless you’re testing send time issue both emails at the same time.

Make sure your email split test is statistically significant by testing it with enough recipients for the results to be viable. 

Let your test run for long enough to get meaningful results. You need to allow for the fact that not everyone opens an email as soon as they receive it. 

Use the data to inform your future email campaigns.