What is A/B testing?

A/B testing is the act of running a simultaneous experiment between two or more pages to see which performs or converts the best. Despite the name (A/B testing), the experiment can be conducted with as many pages as desired.

Once you have decided what to test on your landing page (e.g. headline, call to action, photography, adding a video etc.) – you can create new versions (or variants) of your page to enter into the experiment.

An example of landing page A/B testing

Naming Conventions

Champion page
When you complete a test, you decide upon a winner (the page with the best conversion performance). This is the champion page.

Challenger page(s)
When starting a test, you create new versions (variants) to challenge the existing champion page. These are called challengers.

Page variant
This is a term for any new version of your landing page included in the test. The champion and any challenger pages are all variants.

Assigning traffic weight in an A/B test

Traffic is randomly assigned to each page variant based upon a predetermined weighting – for example, if you are running a test with 2 page variants, you might split the traffic 50/50 or 60/40. Visitors are typically cookied so that they will always see the same version of the page (to maintain the integrity of the test). The main factor that decides how much weight you would ascribe to your page variants during a test is timing – whether you are starting the test with multiple variants at the same time or testing new ideas against an established page.

Starting with multiple page variants

If you are starting a campaign from scratch and have several ideas about which direction to take, you can create a new landing page variant for each idea. In this scenario you would most likely assign equal weight (traffic) to each page. The reason being that you want to treat them equally and pick a winner (champion) as soon as possible. You need to drive a certain amount of traffic through test pages before the results are statistically significant or valid, and as you have no conversion data on any of the pages, it makes sense to begin your experiment from a position of equality.

Testing against a pre-existing page

If you have an established page that you want to try some new ideas out on, you would give your new page variants a smaller percentage of traffic than the existing champion to mitigate the risk inherent with introducing new ideas (which may not perform well).