AB testing is now LIVE

AB testing is now LIVE

Many of our customers have been waiting for AB testing and it’s finally live and can be used on our free plan. You can use subject line AB testing on Bulk/Marketing Campaign type only, not on automations or transactional campaigns.

What Is an A/B Test Campaign?

An A/B test campaign is a campaign with 2 or more versions being tested on one dimension – typically subject line or send time. A multi-variate test is a test with more than 1 dimension, for example if you are testing 2 different subject lines with 2 different versions of your message body you have 4 total versions and a multi-variate test, to determine the best combination of subject line and message body.

Why Use AB Tests?

AB tests can help you optimize your campaign engagement, by determining best time or subject line to use. If you have a large list (over 50k) you can test a subject line on 20k and then use the best one on the rest of your list, improving your engagement overall.

You can setup AB test to determine:

  1. Ideal send time for your brand or list – test early on, then use that knowledge forever!
  2. Best subject line
  3. Best converting CTA in your email body

Besides producing better results (sales, conversions), higher engagement helps improve your future inboxing.

How to Setup an A/B or A/B/C Test on a Subject Line

Create your bulk/marketing campaign the same way you usually do, for example by copying and modifying your older campaign.

Determine how many variations do you want to test and click on the A/B Test button next to the subject line label:

 

AB testing is only available on bulk campaigns, not on transactional or automations. You can test up to 3 subject lines on a single campaign

TIP: You would need a good list size or high engagement to get statistically significant results.

A winner is selected at least 2 hours (or later) after campaign is sent, to allow for meaningful data collection, so you will only see raw data for the first 2 hours after a campaign is sent. 

AB test results

The winner is chosen based on open rate.

AB test results

That’s it. If you are a BigMailer customer you can reach out for assistance with this or anything else via chat 7 days a week.

If you want to AB test on send time, please refer to this article for detailed instructions.

Not a BigMailer customer? Why not try BigMailer for free, with no features locked? We are sure you will love it.

 

High complaint rate on Feb 20 for Hotmail, MSN, Live

High complaint rate on Feb 20 for Hotmail, MSN, Live

Many customers who sent campaigns on Feb 20 reported high complaint rates. Upon further review of engagement by domain reports we are able to confirm that the complaint rates went up significantly for these email domains – hotmail.com, hotmail.co.uk, msn.com, live.com, live.co.uk

Why did this happen?

It appears there was a change in the Hotmail algorithm that determines what email go to spam folder vs. inbox. As a result, 3 things happened:

  • A very high volume of spam is been seen in the Inbox.
  • Regular senders inboxed more of their emails than they did historically, seeing higher open rates and as a result higher complaint rates.
  • Some subscribers had to mass move emails to Spam folder or marked large batches of emails as Spam, likely marking some emails as spam without high intent.

What should you do?

Consider using segmentation and exclude high complaining domains in the next 1-2 days to lower your overall complaint rates, whether you are using Amazon SES and worried about suspension or just want to avoid getting caught in the crossfires.

As always, to keep complaint rate low, follow these email deliverability best practices:

  1. Make your unsubscribe link east to see/find, don’t make the text small
  2. Consider lowering your frequency if your complaint rates are high.

To ensure you maintain high deliverability and inboxing follow best practices. If you think you already do, please review this article on advanced deliverability and inboxing tips.

If you are a BigMailer customer and need assistance or would like an audit of your account please reach out to us via chat.

We will be posting updates here if you want to know when this issue gets resolved.

How to setup an AB test on an email campaign

How to setup an AB test on an email campaign

UPDATE: Subject line AB testing is now live. You can use this article to setup other type of AB tests, for example on send time.

What Is an A/B Test Campaign?

An A/B test campaign is a campaign with 2 or more versions being tested on one dimension – typically subject line or send time. A multi-variate test is a test with more than 1 dimension, for example if you are testing 2 different subject lines with 2 different versions of your message body you have 4 total versions and a multi-variate test, to determine the best combination of subject line and message body.

Why Use AB Tests?

AB tests can help you optimize your campaign engagement, by determining best time or subject line to use. If you have a large list (over 50k) you can test a subject line on 20k and use the best one on the rest of your list, improving your engagement overall.

You can setup AB test to determine:

  1. Ideal send time for your brand or list – test early on, then use that knowledge forever!
  2. Best subject line
  3. Best converting CTA in your email body
  4. A combination or all of the above!

Besides producing better results (sales, conversions), higher engagement helps improve your future inboxing.

How to Setup an A/B or A/B/C Test

Step 1: Setup up your A/B or A/B/C segments

Determine how many variations do you want to test – that’s how many segments you need to split your list into. We suggest to define reusable segments using “starts with” condition on one of your custom fields, for example first name.

Random segment example

Then, create a segment that’s basically “not A” to ensure you don’t make a mistake with rule overlap. The example below shows a segment C that’s basically doesn’t match ether A or B, and segment B would be setup similar to A but with different rules.

Random segment B

TIP: Make sure each segment has at least 10,000 contacts in it, so your experiment results are statistically significant.

Step 2: Setup your campaign variations.

Create your bulk/promo campaign, then create copies with your variation – subject line, preview, or message body. Assign a different segment to each campaign, see example in the screenshot below.

AB segment assignment

If you are testing on send time, then your campaigns will be identical, but will just be scheduled to send at different times.

Manual AB test example

Note: The example above is for a B2C list that’s 60%+ Gmail, so those landed in Promotions tab.

TIP: You could create a test using small parts of a larger list, then use the results for the remainder of your list. 

Step 3: Start Testing (almost)

Reach out via chat and request to update your account to allow you to send more than 1 campaign at a time, if you want to test on content and send your campaigns at the same time. By default, accounts have 1 queue and campaigns go out sequentially. If you are on an Agency plan, you already have ability to send concurrent campaigns.

That’s it. If you are a BigMailer customer you can reach out for assistance with this or anything else via chat 7 days a week.

Not a BigMailer customer? Why not try BigMailer for free, with no features locked? We are sure you will love it.