May 25, 2017

When Incremental Won’t Cut It: The Case for Disruptive Experimentation

Leave it to Amazon, Google and Facebook to make A/B testing sexy. Test and learn has immensely increased in popularity throughout the last decade, and thanks in large part to these companies so has the perception that thousands of high fidelity, small A/B tests are the norm.

These tech behemoths all have a few common, relevant characteristics: lots of traffic (hundreds of millions of sessions each month) paired with plentiful and readily available engineering and analytics resources. They all care deeply about high fidelity data, understanding exactly why something was better and being able to move every dial on the board.

These companies run a lot of experiments quickly, including many multivariate tests. And for every little change, they know what moved the needle, what didn’t move the needle, for whom and why.

While the market frequently espouses the virtues of this high volume, high velocity testing, here’s a dirty little secret: the vast majority of companies are not Amazon, Google, or Facebook.

Most companies don’t have a huge front-end development team. They don’t have an army of analysts or UX designers, or even the traffic needed to get to confidence for a large number of small experiments.

There’s a reality about how many tests you can run and the frequency at which you run them. The biggest limitations to a testing program with high volume and high velocity are that:
1) You have to have a certain amount of traffic.
2) You have to have a certain amount of resources.

Moreover, companies that are feeling significant market pressures have a different map of risk-reward than the darlings of the testing world. Time is not on their side in some cases. They don’t have enough resources, and they don’t have enough traffic. So this idea of “Let’s just test thousands of things” is simply not feasible.

Let’s Get Real

Some purists shame the idea of BIG A/B tests that are not run as multivariate tests (MVTs), and similarly shame tests that gang multiple hypotheses into a single test. These are viewed as muddled tests where you might identify statistical difference but you won’t really know what aspect of the change drove the difference. Purists like MANY small tests.

And let’s get real for a minute: the purists are partially right. A higher risk, lower fidelity testing program is not ideal. Ideal would be that you have limitless traffic, limitless resources and that you can run an enormous number of experiments constantly.

But if you can’t do it, you shouldn’t aim for it, right?

So if this uber sophisticated testing is off the table, what does that leave you with? Is it only iterative, incremental change? While there is certainly value in this kind of experimentation, it’s not always enough.

You’ve likely heard the adage “Fail Fast”, but sometimes you have to fail big fast. That is, when you have a big hairy goal looming and limited resources, sometimes you have to quickly make a more impactful change.

Your Incremental Improvements Aren’t Cutting It

If you’re tasked with a 30% increase in conversions and for the last two years you’ve been flat or growing at 2%, you’re not going to get there iteratively. You’re going to have to take bigger swings because you don’t have the luxury of iteratively achieving your target. You’re going to have to disrupt.

When you go from experimenting for iteration to experimenting for either disruption or for enterprise change, A/B tests have the potential to explode into real innovation.

This type of experimentation, which we at Clearhead call DDX (Data-Driven Experience Design), helps you quickly make a big sweeping change. We start with an initial “big swing” disruptive change for a specific portion of the site, and from there we continue to iterate and optimize, reading the signals and learning from our experiment along the way.

Without question, there’s bigger risk with this approach than with typical iterative testing, and you also won’t gain as much clarity on exactly what moved the needle. But if incremental changes aren’t cutting it, sometimes disruption is a necessity.

The Difference Between Failing Fast and Flagrantly Failing Fast

Before we go any further let me make one thing clear: these big swing experiments should absolutely still be grounded in data. You would be doing yourself a disservice to ignore PSM, as we’ve proven repeatedly that focusing your efforts on solving your BIGGEST problems leads to a much higher likelihood of generating ROI.

Even with our DDX (Data-Driven Experience Design) service, we strongly advise usability testing designs or prototypes BEFORE the disruptive A/B test to get early signals to iterate on.

Additionally, by integrating your A/B test data into your analytics data, you can look at how many events or micro-conversions were affected in the experiment and can make smarter assumptions about WHY the test performed a certain way. While this is not the same as an MVT, you can get some of the benefits of data around specific interactions to better read the test results.

The Portfolio Theory of Innovation Disruption

Should all of your experiments be big swing, disruptive experiments? No. Should your entire optimization program consist of iterative tests? Also, no. Rather, I am advocating for a portfolio theory of innovation disruption.

Maybe you don’t have a lot of analysts, but you have a couple. And maybe they have pinpointed some problem areas that are very specific. In this scenario, you should probably run a specific test because you have high fidelity understanding of the problem.

But if time is not on your side and the analyst or researcher doesn’t truly understand the problem, it may be time for DDX. Disruptive experimentation–despite its higher risk profile–should absolutely be in your arsenal and used when appropriate

How does your innovation disruption portfolio look? The ideal portfolio has a combination of quick, small wins and larger, disruptive experiments. If the entirety of your experimentation portfolio’s weight is comprised of smaller, iterative tests, it’s time to diversify.

Iterative Experimentation vs. Disruptive Experimentation: Your Disruption Readiness Scorecard

No doubt it can be difficult to determine when to choose disruptive experimentation over incremental changes. But there are some clear signals to keep an eye out for, which we’ve compiled in the Disruption Readiness Scorecard below. I also invite you to download a printable version.

While this scorecard is designed to help you assess your disruption readiness, it’s more so meant to help you ask the right questions as you explore your options.

Whether you choose incremental changes or a big swing change, don’t worry about the shaming or the ideal. Do what’s right for you. And if you need any help, you know who to call.