What Is A/B Testing?

A/B testing is the act of running a simultaneous experiment between two or more variants of a page to see which one performs the best.

Imagine, for instance, that you want to test your hypothesis that one headline will generate more leads than another. Sure, you could just make the change and cross your fingers. But what if you’re wrong? Mistakes can get costly.

By sending half your traffic to one version of the page and half to another, you can first gather evidence about which one works best before you commit to the change.

Essentially, A/B testing lets you play scientist—and make decisions based on data about how people actually behave when they hit your page.

A/B Testing Terminology

Variant

Variant is the term for any new versions of a landing page you include in your A/B test. Though you’ll have at least two variants in your A/B test, you can conduct these experiments with as many pages as desired.

Champion

You can think about A/B testing like gladiatorial combat. Two (or more) variants enter, but only one page leaves. This winner (the page with the best conversion performance, typically) is crowned the champion variant.

Challenger

When starting a test, you create new versions (variants) to challenge your existing champion page. These are called challengers. If a challenger outperforms all other variants, it becomes the new champion.

Assigning Traffic Weight in an A/B test

In a typical A/B test, traffic is randomly assigned to each page variant based upon a predetermined weighting. For example, if you are running a test with two page variants, you might split the traffic 50/50 or 60/40. To maintain the integrity of the test, visitors will always see the same variant, even if they return later.

The main factor that decides how much weight you would ascribe to your page variants during a test is timing: whether you’re starting the test with multiple variants at the same time or testing new ideas against an established page.

PRO TIP. Keep in mind you need to drive a certain amount of traffic through test pages before the results are statistically significant. You can find calculators online (like this one from VWO) or use tools like Unbounce’s landing page builder to help you run tests.

Starting from Scratch

If you’re starting a new campaign and have several ideas about which direction to take, you can create a variant for each idea.

In this scenario, you’d most likely assign equal weight to each version of the landing page. For two variants, that’d be 50/50. For three, it’d be 33/33/34. And so on. You want to treat them equally and pick a champion as soon as possible. As you have no conversion data on any of the pages, begin your experiment from a position of equality.

Testing Existing Landing Pages

If you have already have a page that you want to try some new ideas out on, it’s usually best to give your new variants a smaller percentage of traffic than the existing champion to mitigate the risk inherent with introducing new ideas.

This will be slower. It’s not recommended that you try to accelerate an A/B test by favoring new variants though, as they’re not guaranteed to perform well. (Remember, A/B testing is all about mitigating risk. Test wisely!)


What Should I Test on My Landing Pages?

Most marketing departments rely on a mixture of experience, gut instinct, and personal opinion when it comes to deciding what will work better for their customers. It sometimes works out, but often doesn’t. When you start A/B testing, you should be prepared to throw all the boardroom conjecture out the window: the data (properly interpreted, anyway) doesn’t lie. It’s worth telling your boss this.

There are a certain number of landing page elements that you can focus on in your testing. The different variations and content that goes into the test are up to you, but which one works the best (whether you like it or not) is up to the customers.

Some of the elements you should consider split testing are:

Headlines

Your main headline is usually a succinct rendering of your core value proposition. In other words, it sums up why anyone would want your product or service. There are many approaches you can try when testing your headline:

  • Try a longer versus shorter headline
  • Express negative or positive emotions
  • Ask a question in your headline
  • Make a testimonial part of your headline
  • Try different Unique Selling Points

Call to Action (CTA)

The call to action is a button that represents your page’s conversion goal. You can test the CTA copy, the design of the button, and its color to see what works best. Try making the button bigger, for example, or make it green for go, blue for link color, orange or red for an emotional reaction.

Hero Shot

A hero shot is the main photo or image that appears above the fold. Ideally, it shows your product or service being used in a real-life context, but how do you know what hero shot will covert for which landing page? Do you go with the smiling couple? Or maybe a close up of the product itself? Experiment and find out.

PRO TIP. Just like your headline and supporting copy, the hero shot is subject to message match. If your ad mentions mattresses, but your landing page’s hero shot shows a rocking chair, then you’ve likely got a mismatch.

Forms

Depending on your business, you might need more than just a first name and an email. If you have a particularly strong need for data, try running a test with many variations of your form at different lengths. This way you can make an informed decision about what abandonment rate is acceptable when weighed against the extra data produced.

Copy

Often the biggest factor is long copy versus short copy. Shorter is usually better, but for certain products and markets, detail is important in the decision-making process. You can also try reordering features and benefits, or making your language more or less literal.

There are lots of opinions on what works and what doesn’t, but why not test it and see for yourself?

Layout

Will a CTA on the left outperform one placed on the right? And does that testimonial video do better if you put it at the bottom of the page or the top? Good question. Sometimes changing the layout of a page can have major effects on your conversions.

PRO TIP. If you want to experiment with layout, move one thing at a time and keep all other elements on the page the same. Otherwise, it’ll be difficult to isolate the changes that work.

Other Methods of Landing Page Testing

If the landing page fundamentals aren’t covered then you can run into issues before you even begin A/B testing. Use a landing page testing tool like Unbounce’s Landing Page Analyzer to make sure that your pages are doing everything they can to convert before you start A/B testing.