A Beginner’s Guide to A/B Testing Your Email Campaigns - BuildFire
Get started

A Beginner’s Guide to A/B Testing Your Email Campaigns

E-mail lists are definitely the best “tool” that businesses use in to leverage their customers in the best possible ways. When you have a growing list of email subscribers that truly gave you the permission to keep them posted with your latest content, products, or (and) services, you are in business!

Still, performing productive e-mail campaigns is not that easy as it looks. You might have a lot of subscribers, but if they’re not opening and clicking through your emails, it’s all in vain. As marketers and businessmen, we’re developing sales funnels with one purpose in mind: sales. It’s called sales funnel because the entire operation is purposed to bring in revenues through sales.


Optimized Emails Lead to Optimized Results

The whole purpose of optimization is to get closer to your desired end results. So if you’re expecting to make 5 sales a day consistently and you’re only making an average of 3, you need to change something. Usually, the first thing you need to look at when something’s not working – your email sequence. Something needs to be tweaked, and it has to be done immediately.

That’s what A/B testing is all about – tweaking your e-mails, creating variations, and changing the perspectives with the purpose of figuring out what works best for your list. Just to be sure that you’ve totally understood what e-mail A/B testing is, here’s another simpler explanation:

A/B testing is the concept of testing variations of your e-mail campaigns to different subsets of subscribers with the ultimate purpose of figuring out which variation delivers the best results. Easy, right?

The testing can vary in complexity, depending on the goals that you initially set. You can, for example, change the subject lines of your emails while sending the same content to different groups of email subscribers. Or, you can complicate things in order to get maximum feedback.

During today’s post, you’re going to learn the basics of e-mail A/B testing. Moreover, in case you’re a total beginner, you will also benefit from a step-by-step guide to establish your own e-mail A/B testing campaigns in a matter of a few hours. Don’t forget to note down the useful insights and tips you come across; they will be useful later!


1. Identify The Problem & Establish The Goals

In order to identify the “bugs” that are dropping your conversions or are simply clogging your e-mail sales funnel, you have to look at 2 important metrics:

  1. Open Rates – this is the metric that statistically shows you how many subscribers from your list are opening your emails.
  2. Click Through Rates (CTR) – this metric represents the ratio of subscribers who click links inside the emails they’re reading. It is calculated like this: CTR = Clicks/Impression. So for example, if the clicks are 5 and the impression is 1000, it means that your CTR is (5/1000 * 100) 0,5%.

These are basically the main influencers for your overall conversions performance, and they’re easy to test. Now…let’s say that many people open your emails, but very few are actually clicking through your links. It’s obvious, you need to tweak something in order to develop your CTR rates.

It’s important to know your A/B testing goals before even beginning the process. Decide which metric you’re focusing on (you can do on both, but not recommended at first) and make the decision final. You can now move on to the next step.


2. Establish What You’ll Tweak

Once you know what you’re aiming for, start analyzing your e-mail campaign and your list. There are some factors that you really need to consider tweaking. The cause-n-effect principle applies here too. Whatever small changes you’ll make on different parts of your emails will surely influence the long-term result that you’re aiming for. Here’s what you should mainly pay attention to:

  • The Subject Line

e.g. “Improve Productivity in 90 Days” vs “The Ultimate 90 Days Productivity Guide”

  • The Headlines of Your Emails

e.g. “Grab this amazing 90% discount today” vs “90% Discount on the best *products*”

  • The Call-to-Action (CTA)

e.g. “Grab a copy now for only 9$” vs “See Features & Pricings”

  • The Structure and the Layout of Your Emails

e.g. Two column vs. single column

  • Which Testimonials You Have to Add

e.g. Some testimonials vs All testimonials

  • The Way You Address the Person

e.g. “Ms. Johnson,” vs “Jane”

  • The Way You Close the E-mail

e.g. “Reply this email and let us know what you think…” vs “Check out more tips on our page”

  • The Body Style Personalization

e.g. Times New Roman, 12 vs Calibri, 11

  • Your Specific Offers

e.g. “Win free shipping” vs. “Earn a 30% discount”

  • The Images You Use

e.g. Images with humans that show emotions vs Images with the product only

Each of these elements is influencing your conversion metrics along the way. For example, your headlines are greatly influencing whether the person keeps reading your offer or not. The call-to-action is influencing your conversion rate directly, as people won’t be tempted to take action and purchase something without enough encouragement.

Before launching the A/B campaign, you need to have a planned testing sequence. Decide which of the components listed above you’re going to test first, how many you’ll test at once, and how you’ll end your campaign. Moreover, this planning process should be connected to the campaign’s purpose (open rates or CTR).


3. Test Every Subscriber or Just Some?

Generally, you’ll want to test your entire list. If you want to get the most accurate impression of how your audience performs to your new email campaign, you need to get in touch with all of your subscribers. However, there are some special situations in which you shouldn’t test your entire list, but just parts of it:

In case you’re offering a discount or limited time offer, giving the prize to everyone would no longer make it a prize. Decide which subscribers receive it, and start sending different emails to different segments of your list. Make it a couple hundred subscribers, but not more. Once you receive insights, decide upon the best campaign version and only then begin scaling it.

If your list is huge and your A/B testing company charges you per numbers, it might be effective to test the biggest possible sample. Make sure that every subscriber is random in order to improve the accuracy of the results.

Looking to do something bold with your campaigns and afraid that it might hurt you later? That’s the case when you’re restricting the tweak to just a limited amount of subscribers. If it goes wrong, you can simply go on with what worked until now or something else.


4. Analyzing the Results

Obviously, the two things that you might be tracking would be your open rates and click-through rates . After all, that is why you’ve begun testing in the first place. These are the components that stay “inside” the e-mails. Yet, there’s another important element that needs to be analyzed. It is present both “inside“ and “outside” your e-mails, and it’s called the conversion rate.

Many people would argue that you can’t analyze the conversion rate simply by testing e-mails. That is not true. Even though the conversion rate could be severely influenced by your website design, UX, or by your marketing strategies, the e-mail campaigns you’re holding are probably the most important part of the conversion process. Therefore, neglecting the conversion rate while analyzing your A/B testing results would be a serious mistake.

Start tracking the conversion rates resulted from your split testing campaigns. You’ll often notice that some campaigns lead to a better CTR but have lesser conversions. The end goal is to make sales, not to get clicks. You can use some A/B split testing tools that’ll help you with the analytics process. Keep reading, as we’ll cover that soon.

Important Tips How to Make A/B Testing Better

1. Split Your Subscribers into Groups (Randomly)

In order to receive a more precise feedback, you need to split your list into random groups of people. Usually, if you’re using an A/B testing email service, you should be able to do this with the help of the app. Otherwise, you need to manually segment your list into these groups which are also called samples.

So, take a random number of random subscribers and create Sample#1. You’ll then do the same with one more group of subscribers and you get Sample#2. One way to do this is to export your email subscribers into Excel and randomly sort it inside the program. If the random element is missing, coming up with efficient feedback and results is usually harder.


2. A/B Testing Works Best for Frequently Sent E-mails

A/B testing on rare e-mails such as holiday promotion e-mails is not actually recommended. Think about it: you’ll only be able to use the results in twelve months. This is why focusing on frequent e-mails is recommended by most of the professionals.

Frequent e-mails are usually newsletters, new articles posted on blog, alerts, and sales pitches. All of these e-mails are usually sent through an automatic email sequence that keeps rolling until you stop it. If you make a small change in just one frequently sent e-mail, you might improve your chance of converting every new subscriber that goes through the sequence.

But how do you test it? Evan Miller gave us a very powerful tool that helps us calculate the minimum sample size for each group of subscribers. Check it out here.


3. Think Out of the Box

If you want to accomplish something above average, you need to think above average. Whenever you’re creating new variants for testing your list, you need to step out of your thinking commodities and try to come up with something unique.

Nowadays, consumers are already sick of seeing the same things over and over again. Coming up with something original will often help you capture more attention and therefore improve your campaign’s performance. A good advice would be to be bold and have the courage to introduce innovative elements in your testing campaigns. The bigger the difference between e-mails is (bold changes), the smaller your sample dimension can be.


4. Establish the “Statistically Significant” Value

Let’s say that your A/B testing campaign covered 50.000 subscribers. Sample #1 opened your emails 6000 times, while Sample#2 opened only 5600 e-mails. Does that mean that Sample#1 is better? Not at all. Quick tip – never rush your final decision regarding the winner of the A/B testing process. There are more factors to keep in mind while deciding which sample performed best in terms of actual profitability:

E-mail marketers named it “Statistically significant” value, and it represents the best metric to look for when deciding which campaign performs the best. There are plenty of A/B significance testing calculators that can be leveraged for free, so you should give them a shot.

Statistical significance cannot be measured after just one A/B testing effort. Before deciding the best campaign version, you need consistent results that repeat themselves at least a few times.


5. Take Action While Learning from Mistakes

When you find a statistically significant result that also matches your previous results and suppositions, you might be onto something. If you’re sure that you got the result that you’ve wanted and the result is positive, start making the change.

You will now have an optimized e-mail performance that’s based on stats rather than suppositions. You’ll come across many challenges during your journey. You’ll also commit many mistakes. The secret to A/B testing mastery is to learn from your mistakes and not repeat them again. Consistent practice leads to mastery, and that’s what you’re aiming for in case you’re really looking to improve your e-mail marketing performance.


6. Use Email Marketing Automation Tools

We’re living in a digital era, where everything starts to become automated. Don’t forget that the internet appeared just 25 years ago. As a businessmen or marketer, you must stay up to date with the newest trends and practices and keep up with the marketplace.

It’s essential that you bring some aid to your efforts. Automation digital tools are everywhere on the web, and they’re luckily available for almost every possible problem. For A/B testing solutions, certain software can make your life easier:

As you probably notice, many of these services are incorporating the entire “e-mail autoresponder” option. Therefore, you can opt for the tool that you believe to be best for your campaign. Use it to ease the entire A/B testing process and automate time-consuming tasks.

That’s how efficient CEOs and entrepreneurs deal with their successful businesses: they delegate and outsource everything they find difficult to perform on their own. This gives them time & energy to focus on the important aspects of their businesses – the aspects that have to be taken care of personally and personally only.



A/B testing is definitely the best way to create an iteration cycle that will gradually improve your marketing and sales performance. E-mail campaigns are always complex. It’s never the same problem or the same solution.

If you’re willing to spend some time to analyze, tweak, and then change your sales funnels, you’re likely to improve your open rates, click through rates, and eventually conversion rates. Performing a successful e-mail, A/B testing campaign isn’t simple at first.

You might get overwhelmed by a number of details and insights you’ll receive. Still, if you are persistent, nothing’s going to stop you from optimizing and mastering the sales funnels you’ll build or come across in the future.


About the Author: Sasha Woss is a content marketer at BestEssays and passionate guest blogger. When she’s not researching ways to make you better at digital marketing, she finds herself at foreign countries or cooking classes. Follow Sasha on Twitter.

Phone app with BuildFire
Guest Contributor

Powerful app builder

iOS, Android & PWA

Unlimited customization