A/B Testing Lessons Learned: 4 Marketing Case Studies

By , July 16th, 2012 in A/B Testing | 13 comments
Drill this into your head. If you don’t test it, you will never really know what your customers think. Bart gets it, despite being an impudent little brat.

Obsessed marketers should never be 100 percent satisfied with their campaigns – and you shouldn’t be either. If 100 percent of your visitors are converting, what’s left to do? (Well, I’d sit on a beach, but there’s no such thing as a 100% conversion rate).

A/B testing can turn around a marketing campaign from sucky to awesome, and I’ll show you how with the following four examples showcasing how others have done it.

Often an A/B testing example may surprise you; some are quite obvious, but each scenario provides a lesson or two that would have otherwise been missed without testing (at the end of the day the most important lesson is that you understand your customers better).

Lesson 1: Highlight the price point

There’s an old saying that if you have to ask for a price, it’s too much. Much like seeing “market price” as the cost of lobster, many customers are put off by this and will bypass it on the menu, being hesitant to order, as they’re unsure of the cost. SafeSoft Solutions saw dynamic results just by adding a bold, noticeable price image to their ad. In fact, the simple addition of the green image, led to a 100% increase in conversions.

AB Testing Market Dialer
Having no price point drives potential customers elsewhere.
AB Testing Market Dialer
Adding the price point provides value and helps differentiate you from your competitors. For SafeSoftSolutions, just including the image increased conversions by 100 percent. And in this instance, the additional graphic element also draws the reader’s eye path to the bulleted list of benefits.

Lesson Learned

You need to make your customers aware of the value your services offer. The price-point helps potential customers see the value in your services and may be the driving force in converting decision makers. Be sure to know your competition and their promotions, however. If you’re considerably higher priced and offering similar services, adding your price point may not make the most sense –or you may just be overpriced!

Result: For SafeSoftSolutions, just including the image increased conversions by 100 percent.

Lesson 2: Extend Trial Offers

If you’re considering offering a free trial or demo but you’re not exactly sure of the duration of the offer, A/B testing is a great way to find the ideal length. This test by our HubSpot partners showed that increasing their free trial promotion from 7 days to 30 days would also increased their conversions by 110%.

AB Testing Hubspot
Is 7 days long enough to really gauge the success of an inbound marketing software?
AB Testing Hubspot
Apparently, HubSpot’s customers preferred the longer trial period with 110% increase in conversions just by changing to a 30-day trial.

Lesson Learned

Just because it’s free doesn’t mean the offer is compelling enough to drive conversions. You may need to extend the duration in order to boost conversion rates. Ask yourself this: Is 7 days long enough to get an accurate feel for my product? If the answer is no, how do you expect your customers to buy into the offer? Remember, if your offer, the landing page or the product itself is weak to begin with, a free trial might improve conversions, but ultimately won’t boost business.

Don’t forget, once you get the customers to sign up for the free trial, it’s up to you to convert them into brand loyalists.

Result: HubSpot’s customers preferred the longer trial period with 110% increase in conversions just by changing to a 30-day trial.

Lesson 3: Design & Positioning

Sometimes, a simple design tweak may be more palatable to your customer, but unless you run an A/B test, you’ll probably never know if a left-adjusted sign-up box is more aesthetically pleasing than one placed to the right. Matt Sullivan of Inbound Strategy changed their site’s landing page, adding more information and playing with the location of the CTA.

AB Testing Direct Capital
This is a great example of a standard CTA. Features and benefits are clearly stated and it seemed to be working. But, was it working as well as it could?
AB Testing Direct Capital
The right-sided CTA outdid the control page… but only by one  additional conversion.
AB Testing Direct Capital

Lesson Learned

Don’t become complacent. Always be testing! Although the control page was garnering conversions, Sullivan knew that he could make it better through monkeying with the layout – and it worked. Despite the opinions from others in the marketing department, the left-sided CTA won, hands down.

The left-sided CTA saw a 217% increase in conversions compared to the control landing page and variant A.

Bonus Lesson

HIPPOs aren’t always right. Remember: It’s what your customers prefer that drives conversions, and unless you’re not interested in more sales, your opinion doesn’t matter.

Lesson 4: Simplistic vs. detailed

Just for s**ts and giggles, Ben of PlentyofFish created and tested two versions of a CPC ad for EA Sports’ Need for Speed. The first ad was a standard, professional-grade ad that you typically see for a high-dollar company like EA Sports. The second was a sketch done on Microsoft Paint with limited text and a drawing of a race car that even my 5-year-old could do. Much to everyone’s surprise, it was the amateurish sketch ad that outperformed the glossy.

AB Testing Need For Speed

This standard ad for EA Sports was performing well, but could it do better?

AB Testing Need For Speed

To everyone's surprise, this ad performed better than the professionally designed variant. There's much to be said for simplicity.

Lesson Learned

Sometimes, the more simplistic the ad, the more it piques people’s interest; thus, increasing conversions. Never underestimate (or overestimate) your consumers. The hand-drawn, less-than-professional ad tactic won’t work for everyone in every industry. In fact, this concept alone probably won’t work for most. But, it may be worth a test run.

Bonus Lesson

Remove your branding. Another hypothesis surrounding these ads is the variant performed better because it didn’t include the EA sports logo. There has been some speculation that the brand has become less than reputable, drawing more conversions to the unbranded page. If you have any reason to believe your brand name is hindering your conversions, consider removing it and testing. On the flip side, if you’re brand is what sells your product, be sure it’s included.

On another (editor’s note) – if your brand sucks to the poitn where you need to remove it to improve conversions – you are in trouble.


So What Now?

A/B tests shouldn’t just be something you talk about doing; they should be something you do in standard practice. It’s not just a great way to find the right formula for your demographic, but A/B tests also remind you that nothing is ever quite as it seems. Just because you think you have the best landing page, email campaign or ad doesn’t mean your customers will feel the same way.

Minor details sometimes make a world of difference and you’ll never know unless you’re always testing.

– Angela Stringfellow

This is a guest post, all opinions are those of the author.

Angela Stringfellow has spent the last few years of corporate life working in marketing management in the healthcare industry before transitioning her real-world marketing experience to the web. For the past several years, Angela has worked as a consultant with businesses small and large to build comprehensive social media campaigns, blogging and editorial strategies and enhance overall brand reputation and media presence, with a primary focus on Web 2.0 technologies and content marketing.

Comments

  1. This just goes to show the importance of A/B testing. The “simple” Need for Speed ad performing better than the professional one was a bit of a surprise to me. I guess you will never really know unless you test, test, test!

  2. A/B testing is really important to increase conversion. I am using “Unbounce” to design landing page as well as for A/B testing. It is really nice tool. You also have to focus on credential. It build trust in users mind.

  3. Excellent article – I love the quote “It’s what your customers prefer that drives conversions, and unless you’re not interested in more sales, your opinion doesn’t matter.”. Never a truer word said!

  4. chrishall85 says:

    LOL. Love the MS Paint drawing.

  5. While it’s intresting to watch others studies, it’s very rare that anyone actually can use the result of this test.

    I don’t think it’s the right way to tell other people which is “the best way” or what lessons to be learned, because no business is the other one alike. We all have different kinds of visitors, copy, images, trust in brand etc.

    Just because showing the price made 100% more conversions than without – this is not applicable on every website / business.

    I hope you learned my lesson Angela? :)

  6. Awesome post, loving the MS paint ad, I still don’t know why dating sites have such good tech blogs (PoF and OKC)

  7. […] A/B Testing Lessons Learned: 4 Marketing Case Studies – Four interesting A/B tests that marketers might want to consider for their landing pages (and results from companies who already ran them). […]

  8. Lynn Dalsing says:

    Thanks for a great post! It’s always good to have new ideas for things to test on landing pages as well as seeing the results from those who have already tested.

    Angela, we linked up to you from our weekly news round up as well: http://blog.ividence.com/en/archives/news-note-inbox-july-23-2012

  9. […] A/B Testing Lessons Learned: 4 Marketing Case Studies – A/B testing can turn around a marketing campaign from sucky to awesome, and I’ll show you how with the following four examples showcasing how others have done it. […]

  10. Andrew Lolk says:

    I love seeing these examples of landing page tests. You can only create a certain amount of tests yourself, so it’s always great to see others testing out landing pages.

  11. ppc.bz says:

    Great write up. It’s always good to pickup new data and ideas from other peoples tests ( although you should always test it out yourself )

  12. rambo says:

    It is a post useful… Thank you!

  13. […] an example of an a/b test inbound marketing software company, HubSpot, ran in an effort to increase the number of people who […]