Conversion Heroes Part 3: Split Testing – An Interview with Paras Chopra

Conversion Heroes

Conversion Heroes is a series of 5-question interviews with experts in the field of conversion. Subjects for discussion include landing pages, copywriting, conversion optimization, social media conversion, email marketing, organic SEO for landing pages and A/B & multivariate testing.

Today’s Conversion Hero is Paras Chopra

Paras ChopraParas Chopra is founder of Wingify, a company that makes Visual Website Optimizer — world’s easiest A/B testing and multivariate testing tool. His aim with Visual Website Optimizer is to bring the joy of testing and conversion rate optimization to every marketer in the world by eliminating all IT integration issues and by making it dead-simple to use. You can read his personal blog and I love split testing blog. You can also follow him on Twitter @wingify.

Many people have experienced the classic boardroom argument, where everyone thinks they have the best idea. Testing puts this type of conjecture to rest. Todays Conversion Hero is Paras Chopra from VisualWebsiteOptimizer.com who’s going to share some of his insight into the concept of comparative split testing.

1. A/B testing vs. multivariate testing (MVT)

Oli: Let’s start with a question many people seem to ask. What’s your take on when each method is most appropriate?

Paras: The eligibility criteria for each method is traffic of course. You should not attempt to do MVT if you don’t have enough traffic on the site. But assuming traffic isn’t a constraint, MVT works best when you are hyper-optimizing. That is, when your aim is to squeeze the last drop of conversion rate juice from your existing design. On the other hand, A/B testing should be used if you want to test completely different designs and ideas. Ideally, an organization should do lots of MVT tests followed by a few large A/B tests.

Oli: What factors influence the decision when choosing a methodology?

Paras: As I said, traffic, number of design resources available and objectives of testing are main factors which influence the decision. MVT typically requires less design resources as compared to large scale A/B test changes. Moreover, as I said, if the objective is to optimizing existing design MVT (or single element change) is way to go. But if you want to do radical changes on the page (say layout change, theme change, etc.) you will go with A/B testing.

2. Big bang vs. iteration

Oli: People often struggle when they start testing with whether to change their entire page, to test a new experience, or to make small-step changes to a single element. What are the pros and cons of each approach?

Paras: For the starters, I always recommend to start with small-step changes in order to truly appreciate the value of testing. Ideally, they should pick a sweet spot on their page (ideal candidates: call-to-action, headline and image) and optimize that by a simple A/B test. Only once they get the hang of the whole process, they should attempt MVT or large-scale A/B test. That’s because it is easy to get de-motivated or draw erroneous inferences if you are doing a complex experiment without much previous experience.

3. What should I test on my landing page?

Oli: What page elements should people be testing?

Paras: Though the answer to the question will depend on the specific URL and organization, here are some of my favorites:

  • The King: Call-to-Action
  • The Queen: Headline
  • Others: Copy, image, number of form fields, number of steps in funnel, required vs. optional steps, number of elements on page, amount of text on page, layout (left vs. right kind of tests)

[Testing your pricing page] is incredibly risky and almost borders on being illegal.

Oli: Are there things that you shouldn’t test? Because the results will be insignificant?

Paras:

  1. Pricing. It is incredibly risky and almost borders on being illegal. You should never offer the exact same service or product at different price points to different customers. There are subtleties for handling a price test so one should really think this trough before setting up such a test.
  2. Trivial elements on page. Theoretically every part of a page impacts the conversion rate but many elements on the page only start getting influential if you have huge traffic (and patience) while doing testing. For example, you may discover that changing a copyright notice increases the conversion rate from 1.1% to 1.115%. Such trivial changes aren’t worth testing to me (of course there are exceptions, so you should go with your judgment if you really think copyright notice impacts conversion rate).

4. For the love of testing!

Oli: Your blog is called “I love split testing” and your startup VWO is designed to easily facilitate A/B & MVT testing. What was the inspiration behind creating the product?

Paras: The inspiration behind Visual Website Optimizer was to create world’s easiest A/B and multivariate testing software which marketers can use to optimize their websites and landing pages without needing to involve technology department. Unlike Unbounce, which is a beautiful landing page creation software, our aim with Visual Website Optimizer is to enable marketers optimize existing pages. Say you are a Marketing Manager at an E-commerce store and you are looking forward to increasing sales. You would then use our product to optimize pages such as: homepage, product catalog, product details, checkout page and perhaps some landing pages as well.

The motivation for “I love split testing” blog is to create a community around A/B and multivariate testing which discovers the potential of testing (and of course our product) by means of articles, case studies, advice, resources, etc. We now have 10+ case studies on the blog which people immensely enjoy reading and discussing.

Oli: I’m going to put you on the spot here. How much of your own dog food do you eat? In other words, do you run experiments on your own website? And what have you learned from this?

Paras: Oh yes, definitely. At any given moment, we are probably running 4-5 experiments on our website. There are miscellaneous learning’s such as embedding FREE TRIAL along side the pricing table converted much better than having it below the table (separately). Another learning for us was that having a ‘Watch a Short Video’ button on the homepage converted much better than a ‘Sign up for Free Trial’ (probably because visitors want to research the product first – nobody seems to simply sign up right after arriving on the homepage).

5. The surprising results of testing

Oli: Testing can either confirm or refute your design & messaging concepts. What are some of your favourite examples of experiments that produced surprising or unexpected results?

Paras: A recent test was very surprising – in this test it was found out that removing a secure icon from the page actually increased conversions by 400%. Another surprising result was that by simply adding a human photo on a homepage, conversion rate can be potentially doubled.

And as I said, one of the test results on our homepage goes really against the standard advice of having a ‘Signup’ button prominently features on homepage. We found that a ‘Signup’ button actually decreased eventual sign-up’s and ‘Watch a short video’ worked much better because after watching the video, visitors were sure of what they are signing up for. (We had a ‘Signup’ button on the video page, by the way).


Thanks to Paras for being our latest Conversion Hero and sharing his knowledge with Unbounce blog readers.

More Conversion Heroes

Part 1: Roberta Rosenberg on Copywriting for Landing Pages
Part 2: Dan Martell on Social Media Conversion
Part 3: Paras Chopra on Split Testing
Part 4: John Hossack on PPC
Part 5: Chris Goward on Conversion Rate Optimization
Part 6: Cindy Alvarez on Point-of-Conversion Feedback
Part 7: Tim Ash on Landing Page Optimization


Question to readers

What surprising or successful stories do you have regarding testing?

About The Author

Photo of Oli Gardner

Oli is Co-Founder & Creative Director at Unbounce. He's seen more landing pages than anyone on the planet. You should follow Oli on Twitter
» More blog posts by

Comments

  1. Great interview.

    Regarding the ‘Signup’ comment, I worked on one website where we came to plateau in terms of conversions by doing the ‘normal’ things e.g. Layout, Value Proposition, Images and so on.

    During a chance conversation with a customer, we discovered that the word, ‘Subscribe’ on a button could be putting people off.

    We played around with the wording and found that the text ‘Join Others’ worked much better.

    With hindsight this seemed obvious. But this small change gave us another significant gain and forced us to check almost every word on the page.

    We’d forgotten that sometimes the ‘micro-copy’ that you don’t ‘see’ any more can act as a road-block.

    • Oli Gardner says:

      Excellent example Andrew. It’s always best to avoid the generic types of statement on your call to action (such as “Submit”) – but yours has a really strong social flavor to it. It has a comforting and subtle community feel.

      If it was coupled with some strong elements of social proof I imagine it could be a very powerful CTA.

      Thanks for sharing.

  2. Oli Gardner says:

    The most critical point in this interview was surrounding the legality (or lack thereof) with testing pricing pages – and what is and isn’t permissible.

    This is a topic that’s very hot particularly for new startups trying to uncover the correct pricing model and structure as they launch their companies.

    Paras, I wonder if you could elaborate on this point and explain to readers what exactly the sticking points are with regard to legality, and in light of this, what you would test on a pricing page if changing the pricing itself isn’t a viable option?

    • Paras Chopra says:

      Sure, Oli. Actually, I worry about legality of price testing because by law (and even ethically) you cannot offer exact same product or service at different price points. And not just legality, looking from a reputation point of view, if one of your influential users discover that s/he is paying more than the other guy for exact same thing, your company could be in serious trouble. And the reputation loss at that time in un-imaginable.

      Ideally, if you want to do price-testing, you should do an item or product centric testing (as opposed to user centric testing). So, for two similar products (but not exactly same products) price one higher and the other lower and show those fixed prices to ALL visitors. For example, if you sell shoes, find all shoes in a similar price range and then fix different prices to different kinds of shoes.

      Of course, you can’t do such price testing if you have a single product (which is the case with most startups). So, what you can instead do is to offer something tangibly extra in the higher priced version of the pricing.

      The key point here is that you cannot offer the exact same product at different price points. You have to vary specifications of the products so that you have a sound basis to price increase / decrease.

      • Oli Gardner says:

        Thanks for the detailed explanation Paras.

      • From what I understand about the legality (at least here in Quebec), you can charge different prices for the same product (car companies do it all the time) as long as the customer pays the advertised price and you don’t screw up and charge more or less than what was in the pricing page.

        It should be ok because the customer agreed on a price and paid the advertised price.

        However, I would check with a lawyer to make sure before using this technique and I’m not a lawyer so use this at your own risk!

        Obviously, the ethical considerations and potential backlash if users find out probably outweigh the benefits.

  3. Rick says:

    Great information Paras, thanks for contributing here!

  4. Lou Egerton says:

    i read a lot just about this topic in the last few days and i imagine it might be true. Eventhough i think everyone is responsible for himself. No Offense, Just my opinion…

  5. dorianne says:

    It’s definitely true, putting some picture or video on your homepage does increase the readers curiosity of your website that’s why they watch the video first before signing up.

  6. Hkrwotdf says:

    cross with banner tattoo designs,