A/B Testing is Better Than Multivariate Testing

Which test tube has the magic elixir in it? A, B or C?
Which test tube has the magic elixir in it? A, B or C?

Whoa there, easy Tiger! Why would you say such a thing? Well, first of all let me qualify that statement.

Both forms of testing have their individual merits and can be used for different situations. A/B is quick and simple and can tell you which page works better for your target audience. Multivariate can test a multitude of options and variants to come up with the absolute peak performer out of hundreds of potential landing page element combination’s. (However, there are certain restrictions that mean that Multivariate (MVT) isn’t even statistically viable until you have masses of traffic).

Note: if you’re unsure of the differences between A/B Testing & Multivariate Testing you should read this post on the differences for clarity.

The Dangers of Multivariate Testing

Multivariate allows you to test unlimited variations of multiple elements on your landing page. So let’s consider an example test scenario and see where the danger lies.

You’re marketing a popular Las Vegas Hotel and your landing page shows the various amenities the hotel has to offer:

  • Luxury Room
  • Relaxing Spa with couples massages
  • Great shows including world famous burlesque dancers
  • All-you-can-eat buffet for the kids

On your landing page you want to test different variations of headline copy, imagery and call to action. The common thought is that by producing every possible combination of page elements you will uncover the perfect landing page. This is partly true, and can be a very effective method of finding your champion page and then using it moving forward. But it comes with a few caveats that you need to be aware of so that you an make an informed decision.

MVT Danger #1 – Nonsense Combination’s

Because the combination of page elements is created automatically for you in a multivariate test, you will run across situations where the resulting landing page simply doesn’t make any sense. A headline proclaiming “All You Can Eat!” combined with a photo of topless dancers doesn’t convey the right message. Add in a call to action that says book your massage now and you have a conversion cluster*&#!.

Logic dictates that you shouldn’t willingly throw this type of experience at your customers.

Because you need to have a lot of traffic passing through your page(s) to make a MVT statistically viable, it can be a lengthy process. During that time you are exposing potentially thousands or millions of prospects to many of these nonsense landing pages. It stands to reason that some of these prospects would have turned into highly loyal and profitable customers for your business. In some industries they use the term “Whale” to designate the big players – the 20% of customers that represent 80% of your business. And you’ve just lost an opportunity with them.

MVT Danger #2 – You Miss Out on the Learning Experience

Yes, you come out with the best performing page, but what did you really learn? Nothing. By doing everything at once, you missed out on the ups and downs of understanding the behavior of your audience.

A/B Testing, One Tweak at a Time

In an iterative A|B test process, you are designing in context. You can take the time to ensure your page tells a coherent story. Each decision is based on an assumption about your target market, so you can quite quickly validate if your assumptions are correct and make a corrective change if not. Over time, you’ll start to develop a closer relationship and understanding of your customers and this is always a good thing.

MVT might be fine for an optimization agency coming in to aid you with producing a page that converts better, but for me that’s a shortcut I don’t want to take with my own business. I want to know who my customers are and what they respond to.


Both methods are valuable tools, but depending on your role (business owner or an external consultant), you will probably have different notions about what’s truly important. Different strokes for different folks. But for my money, I like the user centric approach that A/B testing allows.

Friday Bonus Landing Page Optimization Tip

7% of men have trouble discerning the difference between red and green. If men are your target audience, there’s a 7% chance of your visitors finding a CTA (button etc) that doesn’t clearly stand out from the design – breaking a fundamental rule of landing page design (using contrast to enhance your call to action). So if you’ve had your arm twisted into testing the button color on your page – think about who you’re serving it up to and if you need to use red or green, use other visual cues to draw attention to it and provide the necessary contrast in the surrounding elements.

Oli Gardner

The Unbounce Challenge

What do you think?

This is a fairly contentious subject, so I’d encourage you to share your ideas and experience to get the debate going.

About Oli Gardner
Unbounce co-founder Oli Gardner has seen more landing pages than anyone on the planet. He’s obsessed with identifying and reversing bad marketing practices, and his disdain for marketers who send campaign traffic to their homepage is legendary, resulting in landing page rants that can peel paint off an unpainted wall. A prolific international keynote speaker, Oli is on a mission to rid the world of marketing mediocrity by using data-informed copywriting, design, interaction, and psychology to create a more delightful experience for marketers and customers alike. He was recently named the "The 2018 Marketer to Watch," in the under 46 category, by his mother.
» More blog posts by Oli Gardner