A Simple A/B Test that Generated a 31.54% Lift in Sign-ups [Case Study]

By , February 10th, 2013 in A/B Testing | 17 comments
conversion lift
Learn how to get a conversion lift using A/B testing. (Image source)

You’re probably used to reading posts that run through a number of A/B test examples. But how often do you get the chance to read an article that gives you an in-depth look at a single case study?

Today I’m going give you a detailed look at a simple A/B test that increased sign-ups on a betting forum by 31.54%. The point is to give a deeper understanding of what goes into conducting a valuable split test that provides you with both insights and lifts.

Background information

Client: Bettingexpert.com – a free betting community where tipsters from all over the world can share betting tips, tools, data, and much more
Page: Home Page Bettingexpert.com
Optimization goal: Increase number of signups/new members

a/b testing example

Page Diagnosis

From experience, I know that it’s not always the biggest changes on the page that result in the biggest lifts. Making strategic changes to mission critical elements will often result in dramatic improvements. Instead of focusing on how much the change impacts the page itself, I’ve learned to focus on how much the change will impact the decisions of potential customers.

So when BettingExpert.com approached me and asked if I could help them increase the number of sign-ups via their home page, I immediately honed in on the most mission critical and prominent page element – the sign-up form itself.

Both the design and placement of the form work well. So at first, I didn’t see any particular need to make any changes here. The form copy however is another story, and I saw a lot of potential for improvement.

At first glance, the original from copy is pretty decent and does an okay job of clarifying the purpose of the form. The header says: “Join BettingExpert” and the button copy says: “Sign Up+”

However, the form copy doesn’t convey any value whatsoever and does nothing to answer the question: “Why should I fill out this form and give you my email?”

In other words, the form copy lacks both relevance and value – two of the main things to focus on if you want to write web copy that converts.

Form copy in A/B test

A/B Test Hypothesis and Treatment A

When I created the treatment, I focused on increasing the relevance and value communicated by the header and button copy. In other words – I focused on answering the question “Why should I fill out this form?”

The main – and most tangible – benefit of becoming a member of BettingExpert.com is that you can get free betting tips from top tipsters on a daily basis.

Based on experience from similar tests, I hypothesized that I could accelerate the decision-making process of the prospects and increase signups by focusing the form copy on the main benefit.

In my treatment the header says: “Get FREE Betting Tips” and the button copy says: “Sign Up & Get the Best Daily Tips”

New form copy for a/b test

A clear A/B test hypothesis is absolutely essential if you want to conduct clean and valuable A/B tests that provide you with both insights and lifts. To learn more about how to create a clear test hypothesis read this article.

A/B Test Setup

In order to find out whether my hypothesis would hold water and increase conversions, I set up a simple A/B test with the Control Variant (A) and my Treatment (B).

There are dozens of different tools you can use to conduct split tests, but in my opinion there is no reason to make your test setup more complicated than absolutely necessary. For experiments such as this I prefer to use tools that allow me to create variations and set up tests easily without having to mess around with code.

Check out this video to see how to set up a split test in 3 minutes with no knowledge of code.

Test Results and Validation

I ran the test for 9 days and reached a sample size of 13.560 visitors, and 291 conversions. The standard error was 0%, and the statistical confidence was 99%. From the beginning to the end, the treatment outperformed the control variant – at no point did the control take the lead. The Treatment increased signups by 31.54%.

a/b test control vs. treatment

Test validation and statistics are some of the less sexy aspects of testing – nevertheless, they are extremely important because there really is no point in testing, if you can’t rely on your tests results.

When validating your test results there are 3 basic factors that you NEED to look at:

  1. Confidence Level
  2. Conversion range
  3. Sample size

The big problem for most marketers is that they either pay no attention to these three factors, or focus only on one of them.

But you really need to be aware of all three factors in order to perform valid experiments that provide true and lasting value to your online business.

Check out this 10-minute video and get a basic introduction to test validation and determining the statistical significance of a split test:

What I learned

This test did more along the lines of confirming what I’d learned from previous experiments than actually opening my eyes to something new. But here’s a rundown of the main takeaways of the test:

The copy you use in your signup form has direct and measurable effect on your conversion rate. The tweaks described in this article may seem insignificant from a usability or design stance, nonetheless, they directly influenced the decisions of the prospects and generated a serious lift in conversions.

When you ask prospects to do something online, you will automatically start an internal dialogue in their heads. They need to assess whether the potential benefits of what you’re offering them outweigh what they have to do – or part with – in order to get it.

In connection with mission critical elements like sign-up forms, it’s important to clarify the value of your proposition so you can give your prospects a good reason to carry out the conversion goal.

Follow-up Experiments

Just because you’ve conducted one successful experiment doesn’t mean that you’ve found the best solution or that you should stop experimenting. In fact, I usually use the first test as a sort of “warm-up,” setting the scene for more experiments. Moreover, the first test will give you some initial insight and an idea of how much it takes to influence the actions of your prospects.

In this case, I followed up with an experiment where I tweaked both the headline and the form copy – as opposed to just the form copy. This simple exercise in clarifying value and adding relevance resulted in an additional 3.65% increase in sign-ups.

more a/b test experiments

I’m by no means done experimenting with this page, and I’ve conducted several other follow-up experiments that include button size, button color, and a different value prop.

All right, so this was a closer look at some of the formalities that go into conducting a valuable and clean A/B test. Now all you have to do is get started on your own website – 1, 2, 3- GO!

– Michael Aagaard


About The Author

Photo of Michael Aagaard

Michael Aagaard is the Senior Conversion Optimization Consultant at ContentVerve. When he’s not preaching the CRO gospel as a popular international speaker, he spends his time helping clients improve conversion rates in wonderful Copenhagen. Follow him on Google+, Twitter or get his new free ebook: 7 Universal Conversion Optimization Principles.
» More blog posts by

Comments

  1. Tomas says:

    Unfortunately I read your test manytimes before, try ty show some new exemples.

    • Oli Gardner says:

      I think the purpose here Tomas was to did deeper into how the whole test was run. I know I found it insightful. Do you have some good examples you can share? Would love to read them.

    • Hi Tomas – Thanks for your feedback.

      I’ve referred to this case study in two previous posts on ContentVerve.com. But this is the first time I’ve used it on Unbounce.com. So “many times” seems like a bit of an overstatement.

      Oli is right, the idea here was to dive deeper into how the whole test was run, instead of just showing it as an example.

      Thanks!

      – Michael

  2. Gary Bury says:

    Out of interest, if you go to their website now they are not using your version of the signup form.

    I’m not doubting you results but if it produced such a large increase in signups why have they not adopted it permanently, were the sigups lower quality?

    • Peter says:

      Hi Gary,

      bettingexpert.com is one of our sites and I am contact person to Michael. We are very happy and excited about Michaels work. We will adopt the winners permanently when all tests are finished.

      Soon we will do a redesign and will use all the information we get from Michaels testings for making bettingexpert.com even better.

      Best
      Peter

  3. Hi Gary – I’m still running tests on the form ;-)

    So you must have been among the visitors that are segmented to see the Control. Right now I’m testing button color.

    When I’ve found the ultimate combination of copy and design, we’ll implement it permanently on the website.

    – Michael

  4. Hello Michael, thanks for sharing your case study. I’m wondering if you know a WP plugin for implement a nice form like yours.
    Cheers.

  5. As a matter of fact I do ;-) I use http://hybrid-connect.com/
    for my newsletter form. It’s excellent and now features split testing.

    – Michael

  6. Lia says:

    Thank you for your post Michael. I especially think your point about follow-up experiments is important. Don’t stop with that first one, keep on testing!

  7. […] A Simple A/B Test that Generated a 31.54% Lift in Sign-ups [Case Study], unbounce.com […]

  8. I love all of the posts on here about landing page design. This is really helpful as a nonprofit professional seeking ways to enhance supporter conversion to donors. Thanks for putting this out there.

  9. […] We’ve talked extensively about A/B testing. Just for a quick recap, A/B testing is a process where you compare two versions of a webpage, email, survey, or other piece of collateral to see which one performs or converts better. Most of the discussions that we’ve seen look at A/B testing from the perspective of traditional websites. Optimizing your site for mobile has more to do with making sure that your content renders properly across a range of devices. There’s also a big opportunity to attract more customers through this important medium. Here’s our quick guide to mobile A/B testing. […]

  10. […] diagnose the customer’s behaviors through both mails’ response rate. You can check out this useful A/B case study by uBounce which increased signup by […]

  11. CASE STUDY says:

    […] az infógazdag slide komoly kutatásra és megalapozott érvelésre enged következtetni. Egy webdesign-os kísérletben kiderült, hogy már néhány extra szó is jelentős, esetünkben 32%-os emelkedést okozhat a […]

  12. […] boosted signups 31.54% by offering free betting tips, as opposed to just asking visitors to […]

  13. […] With no printed materials to pay for, A/B Testing becomes a lot less expensive to implement on digital copy. Be prepared to test multiple treatments of your call to action to find which ones result in more conversions. For an example, check out this article from Unbounce. […]

x
Get actionable optimization tips delivered straight to your inbox.

You'll learn:

  • What it takes to build successful marketing campaigns
  • Why your landing page design and copy might be working against you
  • How to increase conversions while delighting leads and customers