No Textbook required. Not even a Dictionary.
Here at Torspark, we are firm believers in asking lots of questions. With that in mind, here are the most common questions about A/B testing.
- What is A/B testing?
- What can A/B testing do for your business?
- How does A/B testing work?
- How do you pick the winner of an A/B test?
- How long should you run an A/B test?
- Bonus Question: What do all of those marketing sales-y speak headlines really mean?
A straightforward bundle of questions to ask, but the answers are a different matter. The answers tend to be made more complicated than they should be from a combination of technical terms and sales-y marketing speak. This is why we decided to answer these questions, minus the dictionary, textbook, or marketing translator to understand. Here we go!
1. What is A/B testing?
A/B testing (or split testing) is when you test two variations of something to see which performs better. Usually you test a new version (B) that you think could do better against the control version (A). Or if you don’t have a control version, you could just start testing both alternatives against each other right away.
You can test more than two variations as well. A three variation split test would be known as an A/B/C test. One letter is assigned for each variation, and you could continue on with four or more variations in this manner.
2. What can A/B testing do for your business?
The primary use for A/B testing is to gather conversion data. To understand the importance of conversion data, let’s talk about data and sales funnels for a second.
If you have enough data, you can construct the flow of your sales funnel. This means determining where your traffic is coming from, how many people are clicking through your website, how many people end up buying your product or service, and what path they took to get there.
However, even the best sales funnels leak. Holes in your funnel are where people drop off at each stage. By seeing the big picture painted by the data, you can figure out where the leaks are. Once you know where the barriers to conversion are, you can start to fix them.
A fix can be anything from better headlines and copy to adding (or removing) videos. Sometimes it’s even as simple as changing the color of a button. It really just depends on your customers and what they respond to.
This is where A/B testing comes into play. It is used to test those fixes to your leaking sales funnel. By A/B testing the fix, you can determine if your fix actually works. Or if your fix made your sales funnel leak some more. Or if it didn’t make a difference either way. Some fixes will have a greater improvement than others, while sometimes you will need a combination of fixes to see an improvement.
With some trial and error, you can find the right combination to optimize conversions and minimize leaks in your sales funnel. The right boost to your conversion rate should increase your revenue.
Now you understand the theory for A/B testing, but what does this look like with some real numbers? Don’t worry, you won’t need a calculator.
Let’s say you have a business selling some software. Your current conversion rate is 10%, and your yearly revenue is $500,000. Your business then conducts several A/B tests to optimize conversions in your sales funnel (without changing prices). As a result, you were able to increase your total overall conversion rate by 3%.
You new total conversion rate is now 13% (10% original + 3% improvement = 13%). That extra 3% increase in your total conversion rate represents an additional $15,000 in yearly revenue for your business.
Not a bad chunk of change for running a few tests. Especially when you consider that your business will get this extra $15,000 every year.
What if your business was able to continue to optimize your conversions overtime? So that 3% increase became 5% or 7% as different tests improved different aspects of your sales funnel. Those percentages will compound (just like compounding interest rates), and revenue will really increase.
So…
3. How does A/B testing work?
Let’s say you want to target the number of people signing up for a free demo on your website. You are worried you are missing out on conversions because you think people aren’t noticing the bland button that blends into the background. You decide to perform an A/B test.
The original button is version “A” and the new bright orange button is version “B”. Half of your traffic to the website will see version “A” and half will see version “B”. Some tools will declare the winning version based on the higher conversion rate after a set amount of time or visitors. They’ll even automatically apply the changes as soon as they pick a winner. This is not how you want to declare a test winner.
4. How do you pick the winner of an A/B test?
Declaring a winner sounds simple, but it can get a little tricky because of the multiple factors: sample size, statistical significance, and getting an accurate representation of your business traffic. (Wait, don’t grab your textbook yet!)
Statistical significance is when a statistic is deemed reliable. So a test with a 95% statistical significance means there is less than a 5% chance that the results of the test could be due to random chance. Remember, the higher the statistical significance percentage, the more you can trust in the results of your test.
To achieve a higher level of statistical significance, you will need to increase your sample size (the number of people you are testing). You will also need to increase your sample size the more variations you test. In all honesty, your sample size can never be too big. This is where time and money become factors, which we’ll explore in the next section. But you will need certain minimums to achieve statistical significance.
Sample size and statistical significance alone are not enough to declare a winner when making business decisions. Your test should run long enough for your sample size to accurately represent your business traffic. These elements combined together help you decide how long to run the test for and eventually for declaring the winner and making the right decision.
5. How long should you run an A/B test?
The very simple statistical answer is once a test has reached statistical significance then the test is complete, but that isn’t entirely accurate when trying to make an informed decision. It is at best a half answer and very misleading at worst. You want to achieve a combination of a high level of statistical significance that is indicative of your conversion funnel.
Consider this…
What if you get all of the traffic you need for statistical significance in an hour? Is that one hour’s worth of traffic truly representative of all your business traffic? Maybe yes… maybe no… But the only way you can be certain is to let the test continue to run.
If you declare a winner that is statistically significant but is not representative of your business traffic, you could be implementing a change that actually hurts your business.
For example:
Going back to our version “A” gray button and version “B” orange button, let’s say version “A” reaches statistical significance in one hour. If you peek at the results early, you may see “A” as being the winner. However, if you let the test continue to run, you’d end up with version “B” having a higher conversion rate in the longer term with a larger sample size. In this scenario, the traffic in the one hour was not indicative of the larger conversion funnel trend where version “B” was the true winner. This is why you shouldn’t peek at ongoing tests.
This gets us back to the original question of how long to run a test.
We recommend that A/B tests should be run through a minimum of two online business cycles. This helps to minimize the chance of anomalies and false positives in the data. In theory, the data that comes from those two business cycles should look pretty similar.
How long is your online business cycle? An online business cycle usually ranges from one to four weeks, but will vary from business to business. It depends on your consistent activities for driving traffic to your business, but we’ll dive into that another time. (If you’d like help figuring out your online business cycle, just shoot us an email.)
If you make the front page of a major site like Tech Crunch, reddit, or Hacker News and get an abnormal spike in traffic during one cycle, be cautious when making any decisions. Ask yourself if that traffic really represents your usual business. You should consider running the test even longer to balance things out.
And this ladies and gentlemen is how a statistical answer to a business scenario can be both correct and incorrect at the same time.
Now you have a basic understanding of A/B testing. So let’s spice things up a little bit and get fancy!
6. Bonus Question: What do all of those marketing sales-y speak headlines really mean?
You think you have a handle on this A/B testing thing and then you come across a headline like this:
“We’ll increase your conversions by 30%!”
Fancy Marketer
That sounds great. Where do I sign up?
You have probably seen similar sales-y headlines elsewhere. Maybe you are thinking about that 3% increase in conversion rate earlier. The headline says 30%. Done!
….
Wait a second… Take a closer look at that headline.
It says “increase your conversions by” a certain percentage. Why is this one word such a big deal?
Time to break down the sales and marketing speak and understand what those numbers truly mean. Because the numbers sound great, but they may not mean what you think.
Example: Let’s again say you’re selling some software and your current conversion rate is 10%. We’ll go big with a 30% increase, improvement, or lift on that original 10%. This actually means a true 3% increase to your original conversion rate.
Here’s how it works. The 30% is applied to the 10% conversion rate, NOT the total overall conversion rate. 30% of 10% is 3%. (Here’s how the math works (.3) * (.1) = .03 or 3%.)
In layman’s terms, a 30% increase means your old conversion rate of 10% is now 13%. Huh… that 30% seems a little misleading now, doesn’t it?
Before you get too disappointed, as discussed before, a 3% increase to your total conversion rate can be a huge boost to your revenue. The best part is as long as you have the traffic, your testing options to optimize your sales funnel are endless! The more leaks you patch and barriers you remove to conversions in your sales funnel, the more you will increase your revenue. A/B testing can help get you there.
Note: you also need to remember that a 3% improvement in your conversion rate doesn’t help much if you had to cut your prices in half to get it… But, that’s for another article.
Frankly, we love using A/B testing to optimize conversion funnels. Why? Because little changes can have big results, big changes can have even bigger results, and they all compound together!
What’s your next step? What are you going to test? We’d love to hear about it.
Have a question about what you just read, a question about something else related to A/B testing, or just want to say hello? Shoot us an email!
