What Every A/B Testing Conversation With Your Boss is Missing

What Every A:B Testing Conversation With Your Boss is Missing

Alright boss, we need to have a talk.

Last month, you let me start working on new landing pages, but now that they’re ready, you’re hesitant to let me test them out?

Listen, I know change can be scary.

It’s risky to go outside of what we know and try something different. I get it.

But what if I told you we could test these new pages without risking new leads or acquisition?

Not only do I think this new landing page will help us get a bunch of new leads, but I think we’ll get a lot of people who’ve been thinking about working with with us too.

Seriously, just hear me out.

How do we keep things fresh for RETURNING visitors?

If there’s one thing I know, it’s that your boss is obsessed with converting new traffic to sales.

“POINT MORE ADS AT IT!” “RANK FOR MORE KEYWORDS!!!”, she says. Bring new people to the website and hope those new people convert. It’s always New, New, New.

But what about returning traffic? You know, those people who come back to your website, but haven’t bought anything or even become a lead yet?

If you haven’t updated a high volume page like a homepage in a while, a percentage of your returning visitors may be confused (or bored) with the message you’ve been trying to push. It’s important to test pages not just to convert new traffic, but to show some variation to returning visitors.

Think of the landing page like a television commercial. Keeping it ‘fresh’ is the same reason why brands don’t run the same commercial year round. Did you know that Apple’s famous Mac vs Pc campaign ran for 4 years and had a total of 66 different different spots? (full breakdown of the campaign here)

Even though Apple sold 200,000 Macs after the first month of the campaign being live, and 1.3 million Macs only a few months into the campaign, that didn’t stop them from creating new commercials.

For some reason though, we just don’t think of websites that way and your boss will likely thank you for bringing that to light. Remind your boss that if the messaging of key pages doesn’t update periodically, “on the fence” return visitors may never get that final “Ah-Ha” moment that triggers a purchase.

The web is fluid & incremental growth is still growth. Even if you only see a 1-2% improvement with each test, by the end of the year you’re looking at 12-24% increase, which is never a bad thing.

The test won’t risk half of our lead flow.

If that last section didn’t convince the boss, Here’s something that might blow their mind. An A/B test does not mean a 50/50 test.

Bosses will always be hesitant to sacrifice “what works” in lieu of something that risks precious leads. Minimize the risk by reminding them that your theory can be tested with a relatively small percentage of the overall traffic.

Make it friendly, tell your boss you’ll take them out for drinks if you don’t improve the conversion of the page by at least 5%, and request a sample size of 10-20% of the overall traffic.

Your overall traffic will determine how long your test will need run before you can reach ‘statistical significance‘. If you have a high volume landing page, you’ll get there faster. If your page has very little traffic, your test will be longer, no way around that.

If you’re unsure how long your test should run, our general rule is that each test variant should get at least 100 visitors and the test should run for a minimum of 7 days.

Remind your boss that you’ll also only declare the test page a “winner” if you’re 80% – 95% sure you could duplicate the results. If your boss is wondering just how you can calculate your confidence to an exact percentage, show them the below video that explains how the Chi-Squared test works.

Our visitors are “telling” us these things should change.

Don’t bicker with the boss about what to test first. Instead, employ a few customer development techniques & find out what users actually want.

If you have a customer service department, ask them to share user feedback about the landing page. Email existing customers and ask their honest opinion, use a mouse tracking or eye tracking tools to watch how people interact with key pages, and gently ask visitors through an on-site survey tool for their feedback.

When Unbounce released The Noob Guide To Internet Marketing ebook in 2012, we wondered if the guide should be “purchased” in exchange for an email or a tweet. Oli used the Qualaroo integration to ask visitors which method they would prefer to pay for the download. The results were revealing, and the case study is well worth checking out.

The goal is to arm yourself with as much qualitative data as possible to present a more compelling argument for testing than “I think this looks/reads better”

Talking with your customers and visitors will almost always reveal test ideas that you never planned on, and their feedback is ultimately more valuable than most people within the company (because you know, they’re the ones actually visiting the landing page)

Then, when you bring this information to the boss, you can collaboratively interpret the data and come up with test ideas to solve real problems.

For a great list of customer feedback tools, take a look at this list compiled by Angela Stringfellow.

If At First It Don’t Succeed – Stay Persistent.

The biggest thing to keep in mind is that your boss is always concerned about risk & reward. If the risk looks too high, and you haven’t tied specific reasons to why the payoff will occur, you’ll never win the battle.

Make sure to go into the conversation with as much concrete evidence as possible, and valid arguments as to why you’d like to test your hypothesis. Some of you may have overcome this obstacle already and seen great results.

If you could, please let us know in the comments:

  1. How you convinced your boss to move forward with your test.
  2. The arguments they made before agreeing to let you test.

As we push the internet forward, it’s important that we, as a community, help each other get every level of management on board to make truly spectacular experiences for our customers.

The best way to do that is to share what we’ve learned from being in the trenches.

— Tommy Walker


default author image
About Tommy Walker
Tommy Walker is the founder of WalkerBots Content Studios, a content marketing consultancy for growth-stage B2B startups and enterprises. Prior to forming his own consultancy, Tommy was the Global Editor-in-Chief of the QuickBooks Resource Center, the Editor-in-Chief at Shopify Plus, and CXL before that.
» More blog posts by Tommy Walker