How to Find the Actionable Insight in Every A/B Testing Case Study You ​Read

By , April 24th, 2014 in A/B Testing | 7 comments
case-studies-header

“I’ll have what she’s having” won’t get you too far without A/B testing.

In my last post I talked about 3 landing page optimization mistakes I’ve made that other people should avoid. One point in particular sparked quite a bit of interest, so I decided to dedicate a whole post to expanding on it.

Today we’re going to take a closer look at what you can learn from case studies and why you shouldn’t always take them 100% literally. Moreover, I’m going to give you a number of tips for how to get the maximum value from the myriad of LPO, CRO and A/B testing case studies that are available on the web today.

Case studies are about inspiration – not absolute truth

When you read a case study in which someone got a conversion lift by, say, changing their CTA button color from red to green, it means that the person who performed the test found out that a green button performed better on their landing page.

It does not mean that green buttons will always perform better than red ones on all landing pages forever.

There are a million things that influence what works on a given landing page, and a specific change might not work when implemented in a different context.

Look for transferable principles

My best advice is to take what you can from other people’s experience and use it as inspiration for your own landing page experiments. One of the best ways of doing that is by trying to understand the underlying principle that made the change work and then find ways of transferring that principle to your own landing page.

Let’s look at an example from the real world that revolves around CTA placement.

Here’s an old case study where a landing page with the CTA placed under the fold performed significantly better than variation with the CTA above the fold.

case-studies-CTA

You could conclude that CTAs should always be placed under the fold. But a more nuanced interpretation might be that you should place the CTA where it best complements the prospect’s decision-making process.

The test results in this case study inspired me to conduct further research and more tests. I found that there is a correlation between the complexity of your offer and how much information prospects need in order to make an informed decision. This correlation influences how far down on the page you should ask for the conversion.

Generally I see a tendency that the higher the level of complexity/scrutiny, the further down on the page you should place your CTA.

case-studies-complexity

We can also use the classic red vs. green button test as an example. While a case study might show an example of a green button performing better than a red one, the transferable principle has little to do with red vs. green – it has to do with visual hierarchy.

It’s really all about using contrasting colors that make mission-critical elements like buttons stand out and assume a higher rank in the visual hierarchy so it’s easy for visitors to identify them.

And as this case study from HubSpot clearly shows, green isn’t always better than red.

case-studies-button-test

The red button outperformed the green button by 21% Image via HubSpot.

 

So if you want to get the most out of the case studies you read, think of them as a source of transferable principles that you can tweak and adapt to fit your particular conversion scenario and landing page.

Use case studies to formulate a test hypothesis

As illustrated in the examples above, taking test results literally and jumping to conclusions can backfire and limit both the insight and results you can gain in the long run.

So when you read a case study, make sure to take the time to interpret the data and consider what’s really going. Once you have a clear idea about why the changes worked in the case study example, it’s crucial that you carefully consider how similar changes could potentially increase conversions on your own landing page.

In my experience, the best way of doing that is to develop an A/B test hypothesis stating what you want to change, how the change will affect the behavior of your prospects and what results you expect to see.

It may sound overly complicated, but formulating a test hypothesis can be as simple as filing out the blanks in this template:

case-studies-hypothesis

Here’s an example of how the red vs. green button test gave me inspiration for a test hypothesis that had nothing to do with buttons.

Remember the transferable principle? In this case, the button test taught me that mission-critical elements need to stand out and assume a higher rank in the visual hierarchy.

This turned out to be directly applicable to a landing page I was working on that offered a free YouTube Mp3 converter.

case-studies-mp3converter-landing-page

A simple squint test indicated that the orange font color made the headline fall in with the rest of the landing page design. And I hypothesized that changing the font color to black would make the headline copy stand, increase readability and boost conversions.

Here was my test hypothesis:

By changing the font color of the headline from orange to black I can get more prospects to read and understand the headline value proposition and thus increase conversions. 

When I tested the hypothesis live on the landing page, it did indeed increase downloads by 6.64%

case-studies-test

A solid optimization hypothesis goes a long way in keeping you on the right track. Moreover, it forces you to scrutinize your test ideas and helps you keep your eyes on the goal. If you want to learn more about hypotheses, this article will give you a more in-depth guide.

And the moral of the story is…

Learn what you can from other people’s results, but don’t just assume that you can simply replicate those results on your own landing page.

You can get a lot of inspiration and ideas from A/B testing case studies, but always take the time to build a solid test hypothesis on how and why this particular test idea will help you increase conversions.

And most importantly, before you draw any conclusions, test, test, test whether your optimization hypothesis actually works in the real world – on your specific landing page and target audience.

Any questions? Let me know in the comments!

– Michael Aagaard


About The Author

Photo of Michael Aagaard

Michael Aagaard is the Senior Conversion Optimization Consultant at ContentVerve. When he’s not preaching the CRO gospel as a popular international speaker, he spends his time helping clients improve conversion rates in wonderful Copenhagen. Follow him on Google+, Twitter or get his new free ebook: 7 Universal Conversion Optimization Principles.
» More blog posts by

Comments

  1. Angela Slupe says:

    Great article, Michael. Marketers need to also remember that their audience may not respond the same way as the audience in the case study.

    • Thanks for the kind words Angela! And very good point about target audiences having different reactions. Motivation and context are two decisive factors when it comes to how people choose to say yes online.

      Michael

  2. Sundeep says:

    True! Most of the times having CTA at the top of the website doesn’t help because the user is yet to consume the information about the product and only then comes decision making process.

  3. Vinay Koshy says:

    Hadn’t thought of using case-studies as a potential basis for developing A/B testing hypothesis, so thanks for the insight.

    Testing the hypothesis though should only be carried out if similar changes to your landing page could potentially increase conversions. Otherwise it may be a case of engaging in testing for the sake of testing as opposed to engaging potential customers.

  4. Thanks for the comment Vinay, glad I could offer you some inspiration ;-)

    - Michael

  5. I’ve been trying to get my A/B tests right, hopefully this article helps. Thanks

x
Get actionable optimization tips delivered straight to your inbox.

You'll learn:

  • What it takes to build successful marketing campaigns
  • Why your landing page design and copy might be working against you
  • How to increase conversions while delighting leads and customers