The unGlossary of A/B Testing: Terminology & Tips to Run Better Tests

Remember when you learned your ABCs? You were two years old, and you felt like a superstar. It was your biggest milestone after walking, and now here you are kicking butt as a rockstar marketer (way better than being an astronaut).

Now it’s time to master one of your most important marketing milestones — A/B testing. Luckily, it’s just as simple as learning your ABCs (or close anyway). Read through this post, and you’ll feel better than you did when you learned how to say the alphabet backwards.

A

A/B Testing

Let’s start with the big picture. A/B testing is a research technique that marketers have leveraged for decades (yes, it’s older than the Internet). Start with one page element like your headline or call-to-action, and split your inbound traffic equally so the at each group sees a different version and you can discover which version resonates most strongly with your target market.

A/B Testing Tip
Clearly define what you’re testing up-front as you’ll need numbers to compare between your groups. Do you care about open rates, click-through rates, conversions, none of the above, or all of the above? Hint: there’s more than one right answer.

Anxiety

Fence-sitters are the hardest type of customer. They can’t make up their minds, and unless you use user feedback tools you’ll never learn what’s preventing them from converting. Visitor anxiety is created by barriers on your page, for example: unclear messaging, pricing, long forms or a lack of trust.

Anxiety Tip
Adding social proof, testimonials and other trust inducing elements to your page will reduce this anxiety. As an example, consider a lead gen page that’s asking a lot of questions in exchange for an ebook. Providing a preview lets the visitor see how good you and your content really are, and will be more inclined to break through those anxiety barriers.

B

Big Idea Testing

If your brand is going through a major change (like if you’re changing your website’s user interface), it doesn’t make sense to adjust one small variable at a time. Instead, you’ll want to test everything at once to see the effects of all elements bundled together.

Big Idea Testing Tip
When you’re dealing with big idea testing, you’re running an aggressive experiment with probabilistic successes and failures. Make sure to weigh the risk vs. reward of your test when running such an aggressive test.

Bounce Rate

Your bounce rate is the percentage of people who visit your page and leave immediately, in A/B testing this is also often referred to as abandonment rate. This is a great metric to use when justifying your time spent on A/B Testing. If your bounce rate is high, no matter what you do to improve your product or service offering, your bounce rate will never decrease until you start making changes (and testing) to your page elements.

Bounce Rate Tip
If you have a high bounce rate and a high time on page you probably have a problem near the bottom of the page, maybe the call to action button. If you have a high bounce rate and a short time on page it means that you have something repelling at the beginning of the page, maybe your headline.

C

Click-through Rate

Click-throughs are the first steps to conversion. Measured as outgoing clicks/unique visits, you’ll want this number to be as high as possible.

Click-through Rate Tip:
Keep in mind that there are some nuances to this number. For one, you’re not directly attributing clicks per person, as a visitor may click more than once. You can also have multiple CTRs per page (hint: for each and every link).

Look for rough numeric ranges to tell the story that you want. It’s unlikely that one user is going to sit there clicking 100 times. Unless it’s you trying to cheat on your test, which would be very, very bizarre.

Call to Action (CTA)

Your CTA is what you want your visitors to do on your page, whether completing a form or clicking through to an ecommerce shopping cart page. It’s the big shiny button that you want to focus all the attention on.

CTA Tip:
Use the tips of conversion centered design to focus people’s attention on your CTA (or in the case of a form, the entire conversion area).

Champion

Champion refers to the control page in your test. The one you are testing against.

Champion Tip
An important thing to note here is that even though the Champion is your main page in the test, as soon as it gets out performed (with statistical significance – or personal gut-based confidence), you need to promote the current leader to take it’s place. It’s just like any championship fight. The winner needs to get all the attention (in this case, the traffic).

Click-through Page

A click-through page is most often used middle man between your ad an ecommerce cart page. It’s purpose is to expand on the intention of the ad and warm up the prospect with enough information that they’ll be ready to click your “Buy Now” button.

Click-through Page Tip
There’s no point in breaking your sales funnel by pushing traffic directly from an ad to a cart without at least a little love. Remember the adage, about kissing before sex. The same principle applies here.

Confidence Level

The confidence level is the probability that the measured conversion rate differs from the champion page conversion rate for reasons other than chance alone.

Confidence Level Tip
If you’re a stickler for data, wait until the confidence level reaches 90-95% – then you will have good data to back up your choice of the winner of the test. Other times you’ll just know in your gut that a page just isn’t working, in which case, just dump that variant and try again.

Conversion Rate

The percentage of unique visitors that come to your page and complete your conversion goal.

Conversion Rate Tip
Never be happy with your conversion rate. Just remember that every page can be better. That’s the whole point of A/B testing.

Control Page

Your control page is your (A) page. The one that existed first that you are going to start running tests against. If you run a test and discover a better performing page, it would be promoted to be the control or “champion” page.

Control Page Tip
It can be a good idea to puke a ton of content onto your control page, so that you can make variants that remove and simplify the content. This will give you an idea of the level of content your prospects need to be convinced to convert.

D

Distraction

The first thing a distracted visitor will do is hit the back button, so don’t let them escape from your page. Make it feel like a nicely contained unit.

Distraction Tip
Often you’ll want to link to other pages or assets, such as a photo gallery or a privacy policy page – or just a deeper explanation of a certain point. At this time, you absolutely do not want them to leave the page. So instead, try to launch lightbox windows that contain your supplementary content. As an example, click the “Privacy Policy” link next to the email address on the form on this page. This way you keep them on your page and remove the risk of them wandering off.

E

Element

A discrete unit on the page: a block of text, a form, a button, an image, etc.

Element Tip
Page elements will typically be the basis for your A/B tests. Start with the most important ones which typically follow this type of order:

1. Headline
2. Call to action
3. Secondary benefit statement
4. Hero shot image or video

Test one at a time for a clean test.

External Factors

Your A/B test can never be a fully controlled experiment. There are way too many external factors that could skew results. Screaming babies. Lightening storms. Internet outages. Or more realistically, inbound traffic that comes from an origin you didn’t expect.

External Factors Tip
If you are doing PPC, you only want traffic to arrive at your page from your paid ads to prevent contamination of your data. To help this, ensure the page isn’t getting indexed by Google etc. so that it doesn’t appear in organic results.

Extra Tip: If you want to send traffic from multiple sources: email, PPC, social – create a different page for each so that you can figure out which channel is performing best. You can also change the content on each page to suit the origin.

Experiment

An experiment is just another way of saying “Running an A/B test”.

Experiment Tip
Don’t make changes to your pages during an experiment. If you come up with a new hypothesis for something that might work, either add it as a new variant to the current test, or choose the best performing variant from the current test, make it the champion and run your new idea against that in a new experiment. Remember to document the reasons (hypothesis) behind all of your experiments.

Eye Flow

This is the concept that by presenting any imagery of people (or even animals) looking in the direction of your conversion goal area will lead you to look there too. It works – we’re like sheep.

Eye Flow Tip
If you are using a video on your page, engage the viewer by looking them in the eye, then when the time is right, look and point towards your form or button from within the video, and ASK them to complete your conversion goal.

F

The Fold

The fold is a term from ye olden days, referring to what you could see on the top half of a newspaper when it was folded. The assumption that it’s the area that gets the most attention.

The Fold Tips
This became a big deal in the web world too with pages being longer than screen heights. Lately though, due to several factors (mouse scroll wheels and larger monitors and screen resolutions) the fold isn’t quite as important as once thought. Here’s an interesting post debunking the myth of the fold. One of the key takeaways from the post is that you should design pages that imply there is more to be explored further down the page.

Form Testing

Forms are the life-blood of any marketer in the business of collecting leads, and how you design and layout your forms (and what you ask for in exchange is critical to your success.

Form Testing Tips
Some things you can test on your forms:

• The number of fields
• The relevance of the fields to what you’re giving away
• The design of the containing element for the form (e.g. a box that encapsulates it)
• Required vs. non-required fields
• The position of the form (on the right or left hand side of the page)
• Adding directional cues that point to your form
• The form header (a description of what you’re asking the visitor to do that matches closely what you say in your
  CTA)
• Placing a privacy policy link next to an email address field to improve trust (make it pop up in a lightbox to prevent people leaving your page)
• Splitting your form over more than one page
• The button copy (your CTA)

Funnel

The sales or conversion funnel is the pathway your prospects follow from entry into your marketing sphere to your end conversion goal. The obvious idea is to pour as many people through this funnel as possible, getting them all to Conversion Land vs. Back Button Avenue.

Funnel Tip
A great way to visualize what could be happening in your conversion process (and develop ideas for testing) is to tell the story from start to finish of what a potential custom goes through on their journey. A Pain Point Analysis is an interactive story plotted on a chart that walks through the experience of the user, assigning positive and negative points along their traversal of the funnel. As soon as you hit a downturn in the experience graph you know you’ve hit a pain point and can use this to focus your next testing hypothesis on improvements.

Usage: Click the “Continue” link in the top left of the page to see experience graph tell the story. It ends with a green “success” hypothesis that is an idea of how to fix the broken experience.

G

Garbage in, Garbage Out (GIGO)

A common example of GIGO in testing would be someone who changes more than one element in one test. In essence you “might” be making smart decisions for each element you test, but you’ll never know which change resulted a the conversion improvement or decline. So you might actually want to call it “Ideas In, No Information Out” (IINIO).

GIGO Tips
Avoid GIGO by ensuring that you’re running a clean test.

Goals

Goals can be categorized between business goals and those in analytics software, which we’ll get to later. What are the goals of your business? Do you even know (beyond making money). If you don’t, you’re likely not marketing correctly and to the wrong people.

Know exactly what you’re trying to achieve, and design your pages with a singular mindset to do just that.

In the image opposite, the guy made some money, but only because he cheated and clicked his own ad.

You want your customers to complete your conversion goals, not you!

Goal Tip
Use congruence: This is the ideas that every element of your page is geared to steering the visitor towards your intended goal. It’s all about focus. Check over your page, and if you’re doing/sayig anything that isn’t “in congruence” with what the visitor should be doing, either change it or remove it.

Gut Instinct

Sometimes you just have to go with your gut.

Gut Instinct Tip
If you quickly see one of your test pages tanking soooo badly that it’s obviously a total failure, get it out of there right away and stop wasting valuable traffic.

Caveat: It’s sometimes worth keeping a little traffic going to that page (maybe 20%) because you never know when you might get some surprise results. I’ve seen it happen.

H

Headline Testing

Your headline is the most critical element on your landing page to test.

Headline Testing Tip
Drayton Bird, perhaps the most experienced direct marketer in the world, suggests spending 80% of your time writing your headline. (Source: KISSmetrics). Why is it so important? Message match, if it doesn’t match what the “clicker” was expecting to see when they made that all important click, then you’ve failed them. Shame on you.

A good exercise is to stick your ads and page on the wall and compare the messaging, then refine it until it’s very clear that they are in concert with one another.

Heatmaps

This is another tool that you can use to summarize user behavior and click patterns on your site. See where people are scrolling and concentrating their visual attention.

Heatmap Tip:
See where people are spending their mouse time on your page, if it’s around your CTA, bravo. if they spend an awful lot of time wandering or focusing on a less important area, you may have written something confusing.

Tools you can use for these types of studies include: Crazy Egg or EyeQuant.

Hypothesis

A hypothesis is a statement aimed at solving an identified problem with your page. It should be written in simple language with a problem and a goal. A good example would be:

Our Test Hypothesis
Will allowing visitors to download the PDF by providing their email address perform better than receiving it in exchange for a tweet? Considering that not everyone has a Twitter account, or is willing to share such information with their followers

Hypothesis Tips
Read this blog post to understand the concept in more depth.

Bonus Tip: Brainstorm with people from different departments to get a fresh perspective: customer support (the front lines), a designer, copywriter and an information architect. You’ll be surprised how much untapped wealth exists in your company.

I

Iterative Testing

A/B tests aren’t a one time deal. You should continuously test your landing pages so that you can make incremental improvements as you continue to move forward.

Iterative Testing Tip:
Document the reasons and results for every test, so when your boss asks you why you did something 6-months later you can say exactly why and what happened.

J

Jumping to Conclusions

Don’t do it. Give your tests enough time to reach a statistically relevant level. The exception being the gut-check mentioned earlier.

Jumping Tip:
Patience is definitely a virtue here. It’s really easy to make a quick assumption that your test is failing and ditch a variant that may make a comeback and surprise you. It’s far to easy to do and you’ll often be mistaken, this is why we test. #fairwarning

Justifying A/B Testing

Justyfing testing to your boss, CEO or client can be a tough business.

Justification Tips
Use these two posts to learn how to convince anyone of the value of A/B testing and conversion rate optimization.

K

Key Performance Indicators (KPI)

Valuable business data is more complex than just a click-through rate or bounce rate. Some of the important KPI’s include:

• Visitor to sign-up ration (acquisition)
• Sign-up to paying ratio (adoption)
• Churn % (retention)
• Average value per customer

KPI Tip:
Read this Onboardly post on vanity metrics for a better understanding of data points that matter most.

Keywords

If you’re running PPC campaigns, one of your primary concerns will be message match. This means you need to include important keywords or phrases in your ads that match closely or exactly with those in the important areas of your landing page (primarily the headline).

Keyword Tip
Don’t build your page entirely out of images. Use text so that the search engines can parse their meaning and give you a decent quality score.

L

Landing Page

In the purest sense, a landing page is any web page that a visitor can arrive at or “land” on. However, when discussing landing pages within the realm of marketing and advertising, it’s more common to refer to a landing page as being a standalone web page distinct from your main website that has been designed for a single focused objective.

Landing Page Tip
Two tips:

1. Your landing page should have no global navigation to tie it to your primary website. The main reason for this is to limit the options available to your visitors, helping to guide them toward your intended conversion goal.

2. Subscribe to the Unbounce blog.

Lift

The % difference between your control page and a successful test variant. Or a British elevator.

Lead Generation

Lead gen is all about fairness. In simple terms, if you want $100 worth of data from someone, give them $100 worth of useful business information. This is known as balancing the “barrier to access” with the “size of the prize”.

Lead Generation Tip
Write great content with original and timely data or instructional information and package it up in a PDF. Then offer it up to people in different ways. In the most recent version we did here at Unbounce, we let people choose to receive the ebook for their email address or by tweeting about it.

M

Message Match

We’ve touched on this a couple of times already, but in case you’re not the type to read a phone book from cover to cover, we’d better start at the beginning. Message Match is a measure of how closely your ad text correlates with the primary test of your landing page. If your ad says “Laptop repairs from $100” and your landing page says “Computer fixed starting at $50” you’re getting too generic.

Message Match Tip
If you’re running any kind of marketing campaign, start by creating the ultimate landing page to sell your product or service, then (and only then) write an ad to lead people to it using language that matches from start to finish. Your landing page should look like a bigger extension of your ad. And if you’re running PPC, you’ll get a higher Quality Score and your ads will be cheaper.

Multivariate Testing

With traditional A/B testing, marketers typically focus on one clearly defined element. Multivariate testing refers to the discipline of testing many variables simultaneously.

When you run a multivariate test you use one page and dynamically supply multiple versions of multiple elements. For instance, testing 3 versions of your headline, 2 versions of your call to action, and 3 button colors simultaneously.

That would give you 18 versions (3-headlines x 2-CTA x 3-button-color) of the page that you are testing. As you can see, with each successive element and variation on an element you multiply the number of versions that you are testing. This means that the amount of traffic required to reach confidence increases.

If you are testing 2 versions of 3 elements you have 8 variations (2 x 2 x 2) and if you have 3 versions of 3 elements you have 27 variations (3 x 3 x 3), adding extra tests adds up very fast.

Multivariate Testing Tip:
Don’t do it unless you are driving a lot of traffic, otherwise you’ll be waiting a long time for meaningful results.

N

Non-Conclusive Results

Most tests fail. That’s just an unfortunate reality, unless you’re a conversion god(ess).

Non-Conclusive Results Tip:
Don’t try to force success when it’s not there. Ambiguity is totally okay. Just begin the process of devising a new hypothesis again and keep on testing.

And don’t panic. Here’s what to do when your test fails.

O

Open Rate

Many email service providers let you A/B test your open rate based on things like subject line & delivery date/time.

Open Rate Tip
Services like MailChimp do this really well, where you get to A/B test your headline etc. for a small percentage of your list, then the winner is what gets sent to the remaining bulk of your recipients.

Oli Gardner

Hmm, what do I say about myself without sounding like a self-congratulatory egomaniac? (FTR It wasn’t my idea to add me in here).

Oli Gardner Tip
Follow @OliGardner on Twitter :) I know stuff about A/B testing and conversion.

P

Post-Conversion Strategy

This is my favorite and probably the most overlooked area for testing. Confirmation pages appear after the completion of a lead gen form or after completing a shopping cart checkout on an ecommerce site. The reason it’s such a hot spot is that it’s the first page a customer sees after converting, meaning that they have invested trust, time, personal data or money with you. So you need to take advantage of this short window of opportunity to try and get a little more out of them.

Post-Conversion Strategy Tips
Things you can test placing on your confirmation pages are:

• Social sharing widgets: As mentioned earlier, by placing them here you are removing the clutter from the main
  page.
• Newsletter subscription: Ask them to subscribe to your newsletter (and explain the benefits of doing so, i.e. what
  they’ll get: tips, advice, discounts etc.)
• Suggested products: Follow the Amazon model, for ecommerce confirmation pages try adding the classic “If you
  liked that, you might like this.”
• A bonus prize: If you are trying to establish yourself as a thought leader by giving away content, consider adding an
  extra free PDF download, it gives them extra incentive to spread the goodwill via word of mouth recommendations.

Privacy Policy

Having a privacy policy is essential for landing pages that have a lead gen form. It builds trust and for PPC it keeps Google from banning you.

Privacy Policy Testing Tip
Test the position. Many people put it in the footer, but I find it’s more successful in providing that sense of trust when placed directly beside the email field i your forms.

Q

Qualitative Research

This is a technique used to gather useful insights from your target market and/or existing customers. It’s based on opinion and needs to be considered carefully (e.g. don’t listen to one person, listen to 20 and develop your insight from the bigger picture.

Tip:
When developing ideas for your next A/B test, use tools like live chat (Olark), or surveys (Qualaroo), so you can figure out where you’re going wrong.

Quantitative Research

This is where you dig through your analytics to find extra insight as to where, when, and why people are moving through your site/page and converting (or not), using purely statistical analysis.

Quantitative Method Tip:
Hire a Google Analytics expert for a day session and get all of your goals, funnels and reports set up, so that you’re ready report to the higher-ups at the drop of a hat.

R

Return on Investment (ROI)

This falls in line with the earlier discussion about justifying testing to your boss or clients, and also the last point about analytics.

ROI Tip
If you need to understand the effect testing and optimization can have on your bottom line, read this post on Finding Your Customer Acquisition Sweet Spot.

S

Segmentation

Marketers are using more sources for driving traffic than ever and it’s causing funnel gridlock. Sending up to 7 different traffic sources into the same page (hopefully you’re using landing pages and not your homepage) makes it impossible to know what’s working and what isn’t.

So how do you figure out which traffic sources are performing and where best to focus your marketing efforts?

Segmentation Tip
By using separate landing pages for each funnel, you are free to test and optimize to your heart’s content. This lets you empower your different inbound marketing team members (the email guy can run his own show – as can the social media girl) to improve their own area without interfering with or interference from the other channels. It can also build a fun and competitive “which funnel is best” type environment.

Social Proof

People are like sheep in many ways. They follow the herd and like to join in with what others are doing. To illustrate this, imagine walking down a street and seeing one person staring up at the sky. You probably wouldn’t give it a second thought. But if there was a crowd of 30 people all staring at the same thing, you’d quickly divert your attention to see what was going on. That’s social proof and it builds trust in your visitors, helping conversions.

Social Proof Tip
You can add social proof to your tests in many ways: Testimonials, endorsement or customer logos, what social influencers have to say about your product. Other types of social proof are follower counts or even adding social sharing buttons to illustrate how popular a page is. More often than not though social follow and share buttons should go on the confirmation page to reduce distraction from your CTA.

T

Traffic

Where is your traffic coming from? How does your paid traffic behave differently from your organic traffic? What about direct visits and social media? And referral visits? Testing different landing pages for each traffic source will answer these questions and improve your marketing campaigns moving forward.

Traffic Tip
A common question that’s asked is “how much traffic you need when running a test?”. If you can run 100-200 visitors to each of your pages, you can often make gut decisions about pages that are obviously underperforming and initate a new test. But generally you’ll want to wait for about 1,000 total visitors and make sure you run the test for a week to cover daily variances.

U

Unbounce

Start shameless plug: Unbounce is the DIY landing page platform that lets you build, publish and test your landing pages without any IT or dev.

Unbounce Tip
Try the 30 day free trial, it makes A/B testing simple and even fun. If you’re already a customer, did you know you can add viewers to your tests for free? They can see stats and previews but won’t be able to mess with your tests.

User Feedback

A/B testing data can answer the question of “what,” but the answer to “why” can be less than straightforward. Complement your numbers with qualitative findings. This means taking a step beyond your A/B test to ask more complex questions.

User Feedback Tip: Check out tools like Qualaroo, UserTesting and SilverBack for quick, reliable, and low-cost usability testing.

Bonus tip: Read these two posts about user feedback.

• 29 Ways to A/B Test Pages Based on *Real* User Insight
• The Top 10 User Feedback Tools for Improving Conversion

V

Video Testing

Using a video on your pages can have a dramatic effect on engagement and your conversion rate.

Video Testing Tip
Test things like autoplay vs. user-controlled play, adding CTAs, the length of your video, how many you use etc. Take a look at this Case Study: Using Video to Lift Landing Page Conversion Rate by 100%

Visitors

A visitor is a unique individual coming to your website or landing page. We often talk in terms of “Unique Visitor” which means each individual, no matter how many times they visit your page, only gets counted once.

Visitors Tip
When you’re building your landing pages take a few minutes to really understand your visitors. If you know they are coming from an email campaign, particular ad group or a guest blog post, try addressing them personally to improve conversions. ie. From your blog -“Hi Blog Reader!”, from a guest post “Special offer for Hubspot Readers”

W

Web Analytics

Even marketing rockstars sing the praises of Google Analytics, a free program that makes it possible to track various features of website performance. While there are other, proprietary programs that offer more advanced analytics features, I find GA (as it’s known) to be adequate – even if it can be overly complicated at times.

Web Analytics Tip
Learn some tricks:

• Using Analytics to Measure Your Landing Page Performance
• Measuring Social Media ROI & Goal Conversions with Google Analytics 5

X

“X”

E[X]periment is about the best I can do here. After all, that’s what this whole post is about, so it’s kinda perfect.

Tip:
Read the Experiment section again.

Y

“Y”

Y should you be doing A/B testing?

Why You Should A/B Test Tip
Read back over this.

Z

Zero Conversions

Wow, nobody likes your product.

Zero Conversions Tip
Before you panic, check your testing tool to see that there is a conversion goal set, otherwise you may be missing a bunch of conversions that are actually happening.

How’s that for a happy ending?

— Oli Gardner