Usability and A/B Testing – A Special Relationship

Website optimization teams are more effective when UX Consultants and AB/Multivariate Testing experts work closely together.

Why is the relationship so important?

There are few who would dispute the importance of testing new website optimization ideas (hypotheses) in a live environment, with real users, using tools such as A/B testing software, before final implementation on your website. But the optimization ideas need to come from somewhere.

And who is better qualified and has a better understanding of user behaviour than a User Experience or Usability consultant?

Hundreds of hours spent watching and listening to users, sharing their pain and delight during website interaction, naturally results in a good understanding of what makes users tick. And as optimizing a website is all about influencing the behaviour of users in line with business objectives, their role in the orchestration of test experiments is really important.

A quick counseling session to heal a broken relationship

A Multivariate Testing (MVT) salesman a few months ago at a London conference said to me:

“We enjoy proving usability consultants wrong!”

Whilst this is quite an extreme viewpoint, it’s clear that not all AB/MVT professionals are sold on the idea of usability. And in a similar way, many of the Usability consultants I have worked with over the years have been just as skeptical about AB/Multivariate Testing.

So in an attempt to build some bridges between these two camps, we’re going to start by airing each others issues and concerns.

Here are the types of comments MVT companies say about Usability:

  • “Usability testing is not statistically significant due to the small sample sizes.”
  • “Participants do not behave naturally in a usability lab when there are cameras pointing at them and a moderator asking them questions.”
  • “It’s so expensive as you need to compensate participants.”
  • “It’s just a few individuals’ opinion – what do they know?”

And here are some comments about AB/Multivariate Testing from the Usability and User Experience camp:

  • “Where do the MVT companies get their optimization ideas (hypotheses)? They don’t seem to be based on evidence or insights?”
  • “How can they create the design variations for optimizing the website if they don’t have User Experience Professionals, Interaction Designers or User Interface Designers? And they don’t do Usability?”
  • “Everything just seems very random; it’s as if they’re just throwing things against a wall and seeing what sticks.”
  • “How could a change to the colour of a button have resulted in huge conversion rate increase unless there was an underlying issue with the button in the first place?”

It’s just a question of language

One of the reasons why Usability and AB/MVT experts struggle to understand each other as well as they should is their use of language. They often talk about the same things in a completely different way. And this language barrier as well as their different viewpoints can again make it a little more difficult to get along.

Usability (and User Experience)MVT (and Analytics)
UsersVisitors
RecommendationsHypotheses
Design solutionsExperiments
IssuesPain-points, blockages, optimization opportunities
User journeys, pathsGoal funnels, fall-out reports
PersonasSegments
Improving the user experience, making the site easier to useImproving conversion rates
Severity of issuesBusiness impact, confidence

The following Wordle Word Clouds are snapshots of the Wikipedia page content for Usability and Multivariate Testing. Try finding the word “user” in the MVT word cloud or visitor in the Usability wordcloud! Also “Design” is so prominent in the “Usability” wordcloud, and yet so hard to see in the MVT wordcloud.

Usability Wikipedia content wordcloud

Multivariate Testing Wikipedia content wordcloud

The future

UX/Usability guys are wising up to the idea of AB/MVT as a very useful tool for evaluating and fine-tuning all the great recommendations which come out of usability testing sessions and other qualitative research such as surveys, remote usability testing etc.

Websites such as whichtestwon.com have definitely helped raise the profile of AB and MVT in the user experience arena. And the fact that so many of the winning variants are due to enhancements to the user experience – just as often as they are due to persuasive design patterns and messaging – again reinforces everything the UX and usability consultants believe in, and makes them want to use the tool.

The fact that tools are now so readily available increases their exposure to the wonders of MVT. Visual Website Optimizer (for testing existing site pages) and Unbounce (for building and testing new landing pages) are so accessible, there is no excuse not to have a go and learn all about them, even if you don’t know an “experiment” from a “hypotheses”!

Check out the Econsultancy MVT buyers guide for more examples.

Testing companies are now showing a greater interest in usability, or at least pretending to do usability. This suggests they are realizing its importance, or at least have realized it’s a good idea to talk about usability to prospective clients to allay their clients’ fears that the MVT house isn’t a user centric company.

Conclusion

Usability and AB/MVT specialists can and should work together. The disciplines are totally complimentary when the individuals from either camp except the limitations of their own tools and learn to recognise the strengths of the others’.

Finally working in silos is not the way forward. After all, we’re all working towards the same goal; to make your online business more successful. Please let me know your thoughts and feel free to disagree with anything I have said. Would love to hear from MVT companies who are usability testing fans? And usability people who like nothing better than split testing?

Chris Gibbins

About chris gibbins
» More blog posts by chris gibbins

Comments:

  1. gbonthron

    It’s simple – good research, UX, design and clear goals = better testing and better results. Not being risk averse also helps.

    Reply
    • Oli Gardner

      Agreed. And you raise a good point re: being risk averse.

      You’ll never get very far in the optimization game if you’re not willing to take some risks. Not all tests will be successful, but that’s all part of the learning process.

      Reply
  2. Nit

    Usability: Who is the target audience, what it needs, and how therefore it reacts to the system.

    Split testing: Delivering various ‘messages’ knowing that one of the presentations (sum of medium, message, value, action asked, etc) will lead to the audience being most positively influenced than the rest.

    Apart from the fact that these two practices evolved from completely different fields, I really don’t see why an expert who optimizes websites would not use a process involving both usability and split testing methods at one or more stages.

    The way I see it. No matter how much you know your partner, you cannot be 100% sure 100% of the time that they will completely enjoy the gift you bought for them. And on the other hand, you cannot keep buying gifts for your partner without ever asking them what they like, and wait until they show that they like one.

    Reply
    • Chris Gibbins

      Great metaphor!
      For some website optimizer’s out there, it’s a case of the fear of the unknown. Especially when the unknown is something outside of their comfort zone – like usability for split-testers or vice-versa. The answer is to just give it a try – and these days it’s so much easier to get started – with accessible MVT software, great landing page tools etc. And even usability doesn’t have to be expensive. Hall testing or Gorilla testing with your potential end-users/customers is a fantastic way to get rich insights into what to test and how to improve your website.

      Reply
  3. Usability and A/B Testing – A Special Relationship | Unbounce | UXWeb.info

    […] much more effective when UX Consultants and AB/Multivariate Testing experts work closely together. Link – Trackbacks Posted in User experience (UX) | Permalink. ← Beyond Boarding Passes: […]

    Reply
  4. Leo McDevitt

    I’m amazed that not all companies are combining usability and A/B multivariate testing. I am usually the cheerleader-in-chief at my company for combining the two.

    We have not seen any benefit in page rank or traffic volume from usability optimization (my term for it) but we have seen a profound effect in conversions. Usability improvements may also be having a positive impact on our backlink portfolio, but we have no metrics to demonstrate this yet.

    Reply
  5. Usability and A/B Testing – A Special Relationship | Unbounce | UXWeb.info

    […] much more effective when UX Consultants and AB/Multivariate Testing experts work closely together. Link – Trackbacks Posted in User experience (UX) | Permalink. ← Are women paid less than […]

    Reply
  6. Usability and A/B Testing – A Special Relationship | Unbounce | UXWeb.info

    […] much more effective when UX Consultants and AB/Multivariate Testing experts work closely together. Link – Trackbacks Posted in User experience (UX) | Permalink. ← An interview with […]

    Reply
  7. Glenny

    What’s up, just wanted to say, I liked this post. It was practical. Keep on posting!

    Reply
  8. 45 Awesome Posts on A/B, Multivariate, and Usability Testing

    […] Usability and A/B Testing – A Special Relationship […]

    Reply
  9. irish birth certs

    I follow ‘which test won’ quite a bit. I have to say that I don’t always agree with their ‘findings’. Often the margins are so close that the tens of thousands of dollars spent testing I would find a waste of money – but do agree that it has to be looked at.

    Reply
  10. Google

    Google

    Always a massive fan of linking to bloggers that I really like but don’t get lots of link enjoy from.

    Reply
  11. adam and eve sex furniture

    adam and eve sex furniture

    […]below you’ll find the link to some web pages that we believe you’ll want to visit[…]

    Reply
  12. RMUTT

    RMUTT

    […]just beneath, are quite a few completely not related websites to ours, even so, they’re surely worth going over[…]

    Reply
  13. SPA-2X1GE

    SPA-2X1GE

    […]Wonderful story, reckoned we could combine some unrelated information, nevertheless genuinely worth taking a search, whoa did one particular discover about Mid East has got a lot more problerms also […]

    Reply
  14. best work from home companies

    best work from home companies

    […]Here is a good Blog You may Uncover Exciting that we Encourage You[…]

    Reply
  15. where to buy fleshlight

    where to buy fleshlight

    […]we came across a cool site that you just could delight in. Take a appear for those who want[…]

    Reply
  16. adam and eve mystery gifts

    adam and eve mystery gifts

    […]the time to read or visit the subject material or web sites we’ve linked to below the[…]

    Reply
  17. women's clothing wholesale in nairobi

    women’s clothing wholesale in nairobi

    […]please take a look at the websites we follow, such as this a single, because it represents our picks through the web[…]

    Reply
  18. Malaysian Escorts

    Malaysian Escorts

    Malaysia best escort and massage agency in KL and PJ

    Reply
  19. Home Entertainment

    Home Entertainment

    […]always a massive fan of linking to bloggers that I enjoy but don’t get quite a bit of link really like from[…]

    Reply
  20. Webcam model jobs

    Webcam model jobs

    […]check beneath, are some totally unrelated sites to ours, nonetheless, they may be most trustworthy sources that we use[…]

    Reply
  21. taxi

    taxi

    […]Here are a few of the sites we recommend for our visitors[…]

    Reply
  22. online slot machines

    online slot machines

    […]please go to the web sites we comply with, which includes this one, because it represents our picks through the web[…]

    Reply
  23. affordable web design calgary

    affordable web design calgary

    This is Global iTech Systems – web design, web development, SEO, ecommerce website company in Calgary

    Reply
  24. Camiones usados de Venta

    Camiones usados de Venta

    […]just beneath, are numerous absolutely not connected web-sites to ours, nonetheless, they may be certainly really worth going over[…]

    Reply
  25. Nipple Clamps

    Nipple Clamps

    […]one of our guests a short while ago suggested the following website[…]

    Reply
  26. valentinesgifts

    valentinesgifts

    […]always a huge fan of linking to bloggers that I adore but don’t get a lot of link love from[…]

    Reply
  27. vagina massager

    vagina massager

    […]check beneath, are some totally unrelated internet websites to ours, however, they are most trustworthy sources that we use[…]

    Reply
  28. cheap iphone cases

    cheap iphone cases

    […]one of our visitors recently suggested the following website[…]

    Reply
  29. Live adult webcam models

    Live adult webcam models

    […]always a significant fan of linking to bloggers that I adore but really don’t get a great deal of link love from[…]

    Reply
  30. desensitizing cream

    desensitizing cream

    […]Every once in a although we choose blogs that we read. Listed below would be the most up-to-date web-sites that we pick […]

    Reply
  31. coffee for kona

    coffee for kona

    […]very few internet sites that transpire to be detailed beneath, from our point of view are undoubtedly properly worth checking out[…]

    Reply
  32. wayne county auto wrecker service

    wayne county auto wrecker service

    […]we came across a cool site which you may possibly delight in. Take a search if you want[…]

    Reply
  33. Detroit Metro Towing

    Detroit Metro Towing

    […]Every once in a even though we select blogs that we study. Listed beneath would be the most up-to-date internet sites that we opt for […]

    Reply
  34. bean co peaberry

    bean co peaberry

    […]although internet websites we backlink to below are considerably not connected to ours, we really feel they are truly really worth a go by means of, so possess a look[…]

    Reply
  35. Credit Repair specialists

    Credit Repair specialists

    […]that is the finish of this article. Here you will come across some websites that we think you’ll appreciate, just click the hyperlinks over[…]

    Reply
  36. best coffee company

    best coffee company

    […]The information talked about inside the article are several of the very best readily available […]

    Reply
  37. try here

    try here

    […]one of our visitors not long ago recommended the following website[…]

    Reply
  38. coffee beans online best

    coffee beans online best

    […]usually posts some pretty exciting stuff like this. If you are new to this site[…]

    Reply
  39. black gold kona

    black gold kona

    […]Wonderful story, reckoned we could combine a handful of unrelated information, nevertheless actually worth taking a appear, whoa did one learn about Mid East has got more problerms also […]

    Reply
  40. Adam and Eve

    Adam and Eve

    […]the time to read or go to the content or websites we’ve linked to below the[…]

    Reply
  41. Best Lubricant for Sex

    Best Lubricant for Sex

    […]Every as soon as inside a though we pick blogs that we study. Listed below would be the newest web pages that we opt for […]

    Reply
  42. canadian passport pictures Maple

    canadian passport pictures Maple

    […]check beneath, are some absolutely unrelated sites to ours, however, they may be most trustworthy sources that we use[…]

    Reply
  43. descargar gratis

    descargar gratis

    […]very handful of sites that happen to become in depth below, from our point of view are undoubtedly nicely worth checking out[…]

    Reply
  44. workfromhomejobs

    workfromhomejobs

    […]that could be the finish of this post. Here you will discover some web pages that we believe you will appreciate, just click the hyperlinks over[…]

    Reply
  45. web hosting provider

    web hosting provider

    […]Here are several of the sites we suggest for our visitors[…]

    Reply
  46. laptop games

    laptop games

    […]please pay a visit to the web sites we comply with, like this one, as it represents our picks in the web[…]

    Reply
  47. rocks-fff jira massager

    rocks-fff jira massager

    […]please go to the sites we stick to, including this one particular, as it represents our picks from the web[…]

    Reply
  48. padangos kaina

    padangos kaina

    […]Here is a superb Weblog You might Locate Exciting that we Encourage You[…]

    Reply
  49. personal training

    personal training

    […]one of our visitors a short while ago recommended the following website[…]

    Reply
  50. free play slots

    free play slots

    […]The information and facts talked about within the write-up are several of the most effective readily available […]

    Reply
  51. 美国 学术造假

    美国 学术造假

    […]that could be the end of this post. Right here you’ll uncover some sites that we think you’ll enjoy, just click the hyperlinks over[…]

    Reply
  52. free download for pc

    free download for pc

    […]Sites of interest we’ve a link to[…]

    Reply
  53. webcam models

    webcam models

    […]below you’ll come across the link to some web sites that we think you’ll want to visit[…]

    Reply
  54. live webcam models shows

    live webcam models shows

    […]The information mentioned in the post are some of the very best readily available […]

    Reply
  55. Women Toys

    Women Toys

    […]below you’ll come across the link to some internet sites that we assume you’ll want to visit[…]

    Reply
Comments