November 5, 2015

Mythbusting the magic and science of optimization

baseball pitch

“3 Tests Guaranteed to Improve Mobile Conversions”
“How to Reduce Cart Abandonment by 50%”
“5 Ways to Increase Email Signups Today”

Sound familiar? Such attention-grabbing headlines dominate the optimization landscape. Implied is that successful practitioners have a set of obtuse technical skills that, if applied correctly, always win. Or that there are secret tricks that ensure more sales or a better user experience. But in reality, these claims are generally disingenuous marketing or half-baked self promotion.

There are many myths (or misconceptions, if we are being polite) when it comes to testing and personalization. But one of the biggest we come across is that optimization is an exact science with predetermined answers waiting to be uncovered. Or, at the opposite end of the spectrum is the insinuation that optimization is magic, a mystical black box that delivers the perfect customer experience.

Certainly statistics, basic math and psychology are employed in testing, personalization and optimization. But, to be totally clear, an experiment that wins for one business is in no way guaranteed to win for another. We’ve tested thousands of changes across a wide range of companies, developing an extensive library of hypotheses and results we can reference. And while patterns certainly emerge, it is very rare to find a test that we can bring wholesale from one client to another, and guarantee it will win. We’ve seen great ideas that are big winners for one client actually hurt key metrics for another. That doesn’t mean it was a bad hypothesis that wasn’t worth testing. But it does remind us that it is hard to solve customer problems in new and statistically superior ways. When it comes to optimization, almost nothing is guaranteed.

After all, that’s precisely why we test, rather than immediately implementing solutions. Every company has a different business model, different SKUs, a different site experience, a different customer base. And that is the entire point of optimization — there is no such thing as perfect or universally true, there are just better ways to solve unique problems for certain segments.

So let’s get real. If there is no exact science or magic to optimization, what does it take to win? What separates effective optimization from dabbling?


At Clearhead, we don’t claim to have a bag of tricks or scientific claims that work for everyone, but we do have a process that stacks the odds in your favor.

This process requires that you are rigorous about using data to identify problems, prioritize hypotheses, establish segments, design and implement experiments, construct data stories and then share them so the benefits are understood and adopted. This process requires being open to change and committed to breaking old habits that resist data-driven decisions. Most importantly, though, this process requires practice practice practice. You must be diligent about repeating the process to gain confidence, both mentally and statistically. By practicing this continuously across all key investments, your organization and your customer experience become optimized.

It’s no coincidence that practice is the term we use for our organization’s departments at Clearhead. Because while our teams have a unique set of skills and expertise (in addition to curiosity, charisma and good looks), their value has been developed via countless reps in the field, working through that process.

Consider a baseball comparison. You can’t become a great hitter just by swinging the bat like Babe Ruth or Ted Williams. You could watch all the videos, read every book, copy each detail of their swing and still not hit a home run off a major league pitcher. That’s because your brain and your physics are different and unique from every other hitter. So you make assumptions and practice techniques until you find the solution that consistently works for you. You go to the batting cage everyday and get in Malcolm Gladwell’s 10,000 hours. And even then you won’t hit a home run every time at bat. The only guaranteed thing is that a rigorously defined practice makes you more likely to succeed and less wasteful in your at bats.

A repeatable process for practice that is adopted and constantly improved upon is what most correlates to winning. If it was magic, those conversion wizards would be billionaires. If it was science, only a small number of PhDs could do it. In either scenario, false expectations have been set that either go undelivered in the case of magic or scare away practitioners in the case of science.

Winning is not just about successful experiments. It is also about decreasing organizational waste and developing a culture of continuous optimization that is most likely to solve customer experience problems. Winning is repeatable and scalable improvement.

So, here’s the honest headline we want to leave you with — optimization is simply skills applied to a method of practice. Practice isn’t one hire or one piece of software. Practice takes time, it takes a team and it takes investment. But the end result is a repeatable, continuous, virtuous cycle of optimization and validation. That is something we can guarantee.