June 10, 2013

“No thank you. We do not believe A/B testing provides lift” and other things we hear while selling optimization services.

As a Digital Optimization Agency, Clearhead spends a lot of time talking to potential clients and working on sales, and like anyone involved in sales, we hear our fair share of “no”.  

Sometimes we are fortunate enough to get additional feedback on why the answer is “no”.  We treasure this feedback because it can give us insight into market perceptions that our potential clients have which we are not addressing in our pitch, and this insight allows us to better address them for next time.  Test, learn, iterate, right?

This is the first post in what I hope becomes a series on Clearhead sharing “Things We Hear While Selling Optimization Services”.

This post finds us talking to a Vice President of Product at a large media company.  This Vice President is a very confident individual, as senior product people often are and often have to be to do their jobs well, and as this Vice President is politely saying “thanks, but no thanks”, he confidently gives us feedback I had not heard before.

“We actually do not believe AB testing provides lift.”

I was caught off guard by this one for a second and managed to mutter something along the lines of “Umm, hmm.  Well, okay.  Thank you for your time and let us know if you change your mind.”

No matter.  The conversation was done at this point, but it got me wondering:

Are clients beginning to view AB testing as some sort of product that provides lift?  Have we as optimization vendors marketed “Guaranteed Conversion Improvements or Your Money Back” to such a point that we’ve now exposed the methodology of testing to the type of performance judgements new software or site re-designs are subject to?

I am starting to think we just may have, and it could get dangerous if we’re not careful.

Okay then Clearhead, so if AB Testing is not something that provides lift, what is it and why should we do it?

The million dollar question.

So for starters, we think of A/B testing in it of itself as just a methodology to collect a data set for analysis.

There’s a number of different methods to collect both quantitative and qualitative data sets.  Some examples include user testing, voice of customer surveys, user research, heat map tracking, good old fashioned web analytics and event tracking, etc..

A/B testing is not providing the lift and it’s not supposed to.  It’s the different versions of the things you are testing that provide the lift…. or not.  A/B testing simply allows you to create an environment for collecting data to measure lift that’s about as isolated from noise (seasonality, day of the week, time of day, timing of marketing campaigns) as you can get.

Why?  Because in well executed A/B/n and MVT tests, all versions being tested are subject to the exact same noise at the exact same time so the noise is cancelled out.

Saying A/B testing doesn’t provide lift is really like saying — “the changes we are testing are not providing lift” or “the changes I am making to my product do not provide lift”.  Or to take it even further – “The changes I am making with the money, resources, and time that my company is giving me are not providing lift.”  Yikes!

I actually believe it’s the cold and humbling truth that most new features and new designs do not provide lift that is driving this perception on A/B testing.  Of course, some do, but like in baseball, batting .300 in product development is winning.

As a product guy, this always hurt, but the cool thing about A/B testing and the answer to that million dollar question is it helps you figure out which changes matter and which don’t FASTER. And that’s invaluable.

With all this said, maybe this Vice President of Product knew all of this when he gave us his feedback, and maybe he thinks he has a better tool to measure and compare his product changes than through split testing.

We’d be all ears…..and taking notes for our next sales meeting.


  • Austin

    3601 S. Congress Ave.
    Bldg. C302
    Austin, TX 78704
    United States
    View on Map
  • Cincinnati

    1311 Vine St
    Cincinnati, OH 45202
    United States
    View on Map
  • London

    14 Grays Inn Road
    WC1X 8HN
    United Kingdom
    View on Map
Map of Clearhead’s location in Austin, TX

Let’s talk.