December 1, 2016

What is “Test and Learn” (and why should I care)?

Clearhead's culture of experimentation

Test and Learn. Culture of Experimentation. It’s likely that you’ve heard these catch phrases. Perhaps you’ve used them or thrown them out there casually, assuming that they are universally grasped or easily recognized. Alternately, perhaps you think these terms are overused and meaningless.

The truth we’ve seen at Clearhead is somewhere in between. For sure, the terms are overused. But that does not mean they are not very powerful and useful concepts. Over the years clients have regularly asked us, “How would you describe what ‘test and learn’ means?” and “Why bother doing it?” and “How do you know if you have a ‘test and learn’ culture?”.

We’ve thought enough about these very valid questions to frame up some answers and help the dialogue graduate from platitudes and vagaries to something that feels real and measureable. Here it goes…

What is Test & Learn?

Test and Learn is an organizational framework and set of processes wherein all changes and investments are understood to be hypotheses. These hypotheses, or possible solutions for an important problem, are tested via experiments.

Based on the findings from the experiments, investments are made in accordance with data and evidence as opposed to opinion. The process is then repeated and improved upon continuously as the organization gets more data.

Test and Learn can apply to all things digital, from advertising to merchandising to design changes to new features and technology.

In reality, Test and Learn is not limited to digital investments. It applies to all business and/or product investments.

Why Test & Learn?

Every change we make everyday to our digital business is an experiment–the outcome is unknown.

Understanding this, Test and Learn is based on the assumptions that we will be more likely to succeed if we:
(a) Know what problems our changes (experiments) are solving.
(b) Know whether those are the biggest problems to solve for our business and customers.
(c) Have confident data to indicate whether our solutions actually worked (versus a control).

Clearhead’s approach to Test and Learn is done through our PSM (Problem Solution Mapping) methodology. Test and Learn is also the core tenet of companies like Google, Amazon and GE, and for good reason (more on that below).

Some of the benefits a Test and Learn strategy yields include:
1: Lower opportunity cost and cost to serve due to the reduction of changes.
Through Test and Learn, we can lower the opportunity cost and cost to serve through the reduction of changes that don’t actually work, or worse, create incrementally new problems. To put it another way, Test and Learn allows us to avoid or eliminate prolonged failures faster.

This benefit could be measured from the rate of incremental product improvement (conversion rate, RPV, member satisfaction, NPS, etc.) compared to the rate of incremental investment in said product.

2: Greater speed to improvement and optimization.
Through Test and Learn we quickly gather data on our digital investments without spending months aiming for perfection. When these experiments pay off, we learn quickly and then expand them from a sample set in our tests to our entire audience to improve financial impact and user experience quickly.

This benefit could be measured by the rate of net beneficial launches or changes or enhancements in a given period as compared to the same, previous periods.

3: A leaner & more evidence based organization.
Through Test and Learn we minimize debate and the internal politics around investments by making it clear that confident data and evidence are the basis for decisions and not opinion or charisma.

This benefit could be measured by the relative amount of time spent in committees and the speed of approval times in comparison to the amount of time researching problems and testing hypotheses.

 Test & Learn in Action

Amazon’s Culture of Experimentation
It’s no secret that experimentation is ingrained in Amazon’s culture. Jeff Bezos, Amazon’s CEO, refers to it extensively in his 2015 Letter to Shareholders.

“To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there,” he explains. “Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten.”

Bezos further describes Amazon’s commitment to a Test and Learn culture by using a metaphor near to my heart: baseball.  (Share my love of baseball? Check out my Digital Optimization as Moneyball post).

Bezos explains, “We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. This long-tailed distribution of returns is why it’s important to be bold.”

Test & Learn at Google
You’re likely familiar with Google AdWords, Google’s proprietary advertising service. AdWords is a wildly successful Google product, to the tune of over $52 billion in 2015 alone. One thing you may not know: When AdWords was first released in 1999, it wasn’t very successful. Google overcame this through a–you guessed it–Test and Learn process.

“Every year we run tens of thousands of search and ads quality experiments, and over the past year we’ve launched over a dozen new formats. Some products we update every day,” Susan Wojcicki, Google’s form SVP of AdWords explained in her article about Google’s Eight Pillars of Innovation.

She continues, “Our iterative process often teaches us invaluable lessons. Watching users ‘in the wild’ as they use our products is the best way to find out what works, then we can act on that feedback. It’s much better to learn these things early and be able to respond than to go too far down the wrong path.”

How GE Encourages Employees to Innovate and Iterate
Test-and-Learn isn’t exclusive to the digital space: GE uses experimentation to create the latest versions of its physical products such as appliances.

According to the Harvard Business Review (HBR), GE encourages its employees to embrace continuous optimization through a program called “FastWorks.” Based on Eric Ries’ The Lean Startup, GE’s FastWorks program rolls out new products to customers incrementally, relying on firsthand feedback to iterate and improve the product with each sprint.

And the results have been staggering. GE’s former VP of Executive Development and Chief Learning Officer Raghu Krishnamoorthy explains, “This new way of working results in better outcomes for our customers, faster. FastWorks is about constantly experimenting, learning and iterating, and the customer being at the center of everything we do.”

For GE, Test and Learn goes beyond the product roadmap. In fact, it has entirely transformed how the company approaches its business as a whole. “In a fast-moving world, we have also realized that annual events are passé. Every operating rhythm has to become more agile, responsive, nimble, and focused. Consequently, we have moved away from an annualized strategic-planning process to a more continuous process of checking on the environment and context and pivoting where necessary every quarter,” Krishnamoorthy continues.

Putting Test & Learn Into Practice

Unsurprisingly, we have entire Practices and Product sets at Clearhead devoted to Accelerating Test & Learn culture and process within our clients.

Interested in putting Test and Learn to work for your organization? Clearhead can help.

  • Austin

    3601 S. Congress Ave.
    Bldg. C302
    Austin, TX 78704
    United States
    View on Map
  • Cincinnati

    1311 Vine St
    Cincinnati, OH 45202
    United States
    View on Map
  • London

    14 Grays Inn Road
    London
    WC1X 8HN
    United Kingdom
    View on Map
Map of Clearhead’s location in Austin, TX

Let’s talk.