[Case Studies] How Failed A/B Tests Can Increase Conversion Rates

By , October 16th, 2013 in A/B Testing | 8 comments
a b testing experiments gone wrone

“Getting a negative test result is the worst thing that can happen when you perform an A/B test – right?”

While many marketers probably would subscribe to the logic behind this statement, experience from almost 400 split tests tells me otherwise. A negative test result will often give you as much insight and learning as a test that generated a major lift.

Sounds weird? Read the rest of the post, look at some real world examples and find out what the deal is.

How changing one word in the call-to-action reduced conversions on a payment page by 26.55%

Here’s an example from a series of tests I conducted on a payment page for one of my clients. This page is the last step in the conversion funnel and every click means money in the bank.

Experience from a vast amount of A/B tests told me that the call-to-action copy itself would have major impact on whether prospects would end up clicking the button, and I had a number of hypotheses that I wanted to test.

One of them was the hypothesis that the possessive determiner “Your” works better than “My” when used in a button. Based on the fact that websites generally speak to visitors in second person singular form, it would seem strange to suddenly shift to first person perspective and use the word “My” in the final call-to-action.

I was certain that my hypothesis was dead on and I basically just performed the test to show the client that I was right. Boy was I in for a surprise!

When I tested live on the website, it turned out that the treatment with “Your” performed significantly worse than the control copy that made use of “My” – 24.95% worse to be more precise.

Negative A/B Test Results

Needless to say I was humbled and taken aback by the result of the test – my hypothesis was totally off, and what I thought would increase conversions had the exact opposite effect.

This made me wonder whether what I saw here was a fluke or if I had actually stumbled upon something that could be used positively. In other words; whether I would be able to generate positive lifts by changing the possessive determiner to My in cases where the CTA copy made use of Your.

I ended up performing this test on a number of websites and landing pages and consistently saw lifts by using My rather than Your in the button copy.

The most dramatic lift was actually in connection with an A/B tests that Oli and I performed on a PPC landing page here on Unbounce.com. In this case, we saw a 90% increase in click through rate when we tested Get my free 30-day trial against Get your 30-day trial.

Negative A/B Test Results

How adding a privacy policy reduced sign-ups by 18.70%

Here’s another great example of a test where I was very confident that my treatment would kick ass. In fact, I didn’t even perform the test to see whether the treatment would generate a lift, I performed the test to see how much of a lift it would generate.

The client here is Bettingexpert.com an international betting community. I was hired to optimize the home page with the goal of getting more potential users to sign up for a membership.

Negative A/B Test Results

I decided to focus on the most critical part of the page – the sign-up form itself. One of the things that struck me early on was the fact that there the form had no privacy policy. Taking the nature of the website into consideration, it seemed quite safe to assume that adding a privacy policy would mitigate anxiety and make more people sign up.

I decided to test a slight variation of a privacy policy that I had come across on a very popular online marketing blog. The policy said, “100% privacy – We will never spam you”.

Negative A/B Test Results

As mentioned, I felt very confident while setting up the test, and I was excited about the prospect of seeing how much better the treatment would perform in real life. Boy did I have another thing coming!

The treatment with the privacy policy reduced conversions by 18.70%. Again I was humbled and taken aback by the test data. The result was completely counter-intuitive but it taught me an important lesson: simply adding a privacy policy doesn’t guarantee more sing-ups, in fact it can seriously reduce your conversion rate.

Negative A/B Test Results

This negative test result really got my mind going, and I decided to run a number of follow-up experiments to learn more about what made this particular privacy policy have a negative impact in the mind of the prospects.

My tests indicated that the word spam had an undesirable effect – even when used to assure visitors that they would not receive any spam. My hypothesis is that by placing the word spam in close proximity of the form, you actually plant an idea in the minds of the prospects; “Oh wait, could they actually end up spamming me?”

After 3 follow-up experiments, I ended up with a winner that worked well and increased sign-ups by 19.47%. The winning privacy policy said, “We guarantee 100% privacy. Your information will not be shared.”

Negative A/B Test Results

I’ve performed privacy policy tests on other landing pages, and the results have been quite similar. Here’s en example from the landing page for my free CRO e-book 7 Universal Conversion Optimization Principles where I saw a 24% drop in downloads when ran the original “I guarantee 100% privacy.” Against “100% privacy – I will never spam you!”

Negative A/B Test Results

And the moral of the article is…

The goal of a test is not to get a lift, but rather to get a learning…” Dr. Flint McGlaughlin, MECLABS

One might be inclined to view some of the case studies in this article as bad tests. But in fact they weren’t bad tests at all because they provided important insights that eventually lead to serious conversion lifts on a number of different landing pages.

Landing page optimization is an ongoing process, and as long as your test results lead to new insights and learning, it essentially isn’t that important whether the initial test results are positive or negative.

Of course hitting a home run in the first swing is easier on the ego. But when you approach optimization as a process – not a one-off opportunity to swing for the fences – you’ll see that stopping at a few bases along the way is often what it takes to win the game.

– Michael Aagaard


About The Author

Photo of Michael Aagaard

Michael Aagaard is the Senior Conversion Optimization Consultant at ContentVerve. When he’s not preaching the CRO gospel as a popular international speaker, he spends his time helping clients improve conversion rates in wonderful Copenhagen. Follow him on Google+, Twitter or get his new free ebook: 7 Universal Conversion Optimization Principles.
» More blog posts by

Comments

  1. Duran Drake says:

    Hi,
    I think this Sign Ups forms are amazing. It is always better to make some minor changes from our resources and that really give a good return.

  2. Star says:

    Great article. I’ve suspected for a while that privacy statements can influence results either way.

    p.s. Coolest clip art ever!

  3. Bertrand says:

    Great read! And a very valid argument towards the approach of A/B Testing – sometimes it’s important to know what doesn’t work in order to determine what will.

  4. Apoorv says:

    Hi Michael

    Would appreciate your expert guidance on the following.
    I am trying to test an idea, by just creating a landing page, without actually building the product. However, I am unable to decide what conversion rate i should consider a cue to build the product and below which conversion rate i should shelve the plan. I hear that the avg conversion for landing page is 5-15%, however I am sceptical that any conversion rate can be measured against this range, as I believe that the conversion rate shoud be a function of (a) how good the idea is (b) design on landing page (c) how clear CTA is (d) is product paid or not ; among other factors. I am unable to figure out, what conversion rate should be considered a cue to build the product. All help and guidance is appreciated. Thanks in advance. Apoorv.

  5. Midas says:

    Great article. I like your quote: “simply adding a privacy policy doesn’t guarantee more sing-ups, in fact it can seriously reduce your conversion rate.”

    People should take heed of that, even experts can’t right all the time. Follow best practise for sure, but test, test and test some more! You’ll always unearth a few gems and ‘gotchas’ along the way.

  6. You might be interested in the case study of one of our recent conversion campaigns: http://localcitymarketing.com/conversion-rate-optimization-services/#guaranteed-conversions

    In short, what we accomplished in 30 days:
    $407,400 – Gross sales resulting from Engagement Marketing Service
    $5,820 – Cost for All Conversions provided to Customer
    1.43% – Percentage Cost Per Sale for Engagement Marketing Service

  7. yeah says:

    i dont care if “my” performs better than “your”, it sounds retarded

  8. Hi Yeah – thanks for your insightful input!

    - Michael

x
Get actionable optimization tips delivered straight to your inbox.

You'll learn:

  • What it takes to build successful marketing campaigns
  • Why your landing page design and copy might be working against you
  • How to increase conversions while delighting leads and customers