October 16, 2013

Digital optimization as Moneyball


I’m not sure if this metaphor is especially current — is the term “Moneyball” a clearly understood and appreciated concept in 2013? And I’m certain that my metaphor is not fully baked. That being said, I’ll posit the following as a simple, accessible way to think about the value of data-driven digital optimization. I’m attached to the “Moneyball” analogy because (1) I have been enamored of Bill James since the age of eight when my parents bought me my first annual “Bill James Abstract” as a veiled attempt to get me to read more, (2) I still hear perspectives on A/B testing and optimization that I liken to outdated, poorly understood baseball folklore and (3) I think the metaphor helps my team understand and explain why we do what we do.

So, how is our approach to digital optimization like Moneyball? Here’s how:

Our singular goal is to help clients maximize the number of quality “at bats” and preserve “outs.”

There are a number of other analogies to be made here – around the risk/reward of falling in love with home runs and of the value of data and underappreciated metrics. But I think the aboves serve as an effective starting point to frame the conversation.

In the simplest of terms, most businesses are driven by their ability to deliver and improve a value proposition in the interest of acquiring and maintaining customers. For digital businesses, this means that all design, UX, product and engineering decisions need to serve a customer desire. The more accurately the company can super-serve this desire, the more likely it is to succeed. In pursuit of this endeavor, the company is constantly making marketing, content, design, experience and product development decisions that I’d liken to “at bats.” They are a chance to step up to the plate and connect with customer desire. The more informed and efficient these decisions are, the more likely they are to deliver value and yield returns.

Every new design, every promotional email, every new site feature every CTA copy line — these are all at bats. When they miss wildly – when they do not connect, hurt performance and don’t lead to rapid learning – they are ostensibly “outs.” Outs are big swings at the plate that do not lead to any customer or intrinsic learning value. Outs drive up operational cost, marketing cost and opportunity cost. Ultimately, there is a finite number of outs that any company can survive. Outs are very costly financially and to brand perception.

Big, costly site-redesigns for the sake of vanity; chasing “features of the day” because they are bright and shiny and the cool kids are implementing them. Major experience or feature additions driven by an executive’s whims. Those are likely outs. Those are Dave Kingman’s. Those are White Sox-era, pre-PEDs, White Sox-era Sammy Sosa’s. They may occasionally connect for home runs, but they ultimately are expensive ways to waste a lot of outs very quickly. They will ultimately bury a team.

Bill James developed the “Dave Kingman Award” (an award you don’t want to win) for hitters whose predominant contribution to runs created was through home runs, largely masking the cost of the low on base percentage and WAR. Players like Kingman, Tony Armas and Ron Kittle were all decorated all-stars, with gaudy home run totals who most SABER-metricians would argue cost their team far more than they delivered. Companies stuck in waterfall development methodologies, driving design and engineering decisions based on the opinions of a select few, are the Dave Kingman’s of digital businesses.


Companies that are using analytics as a reporting device and dabbling with A/B and user testing are further along in the spectrum. They are not Dave Kingman’s. However, unless that data and the validated learnings developed through analytics and testing are driving all key design and development decisions, the companies are still wasting a huge number of precious at-bats. These are more evolved businesses. They exhibit superior top-line stats, but they still strike out far too often. Think of Reggie Jackson or Jim Thome as a point of comparison – extremely talented hitters who were far too willing to eschew data and situational hitting for big swings (and misses)

Those companies that have developed cultures wherein data, experimentation and validated learnings are driving all key product design and development decisions are the ones maximizing their at bats. They are scoring through a lot of walks, singles and doubles. They rarely strike out and they occasionally hit one out of the park when the count is in their favor. They understand their customer, the product and their market through the benefit if many validated (and invalidated) hypotheses. These companies are the Ted Williams’ (full career), Joe Morgan’s (mid-1970s) and Rickey Henderson’s (1985, 1990) of baseball.

So, how do you maximize at-bats? You employ data for situational hitting? You understand the pitcher’s strengths and weaknesses. You take pitches. You choke up with two strikes. You understand that singles and doubles lead to runs. You understand that a walk is an opportunity to get more data and postpone future outs. You swing for the fences only when you are most likely to “guess right.” OK – I am mixing metaphors now. But you get the point. The idea is that data driven continuous experimentation, iterative changes, targeting and testing allow for more “at bats” – attempts at connecting with the unique customer segments and delivering incremental value. These at bats don’t offer the undeniable appeal of the monstrous three run homer (sorry, Earl Weaver) but they lead to more data, smarter swings and a greater likelihood for solid contact.

I’ve likely belabored this metaphor enough and should bring the big point home. Digital Optimization is about data-driven at bats. Digital Optimization isn’t about home runs. People are justifiably seduced by tales of 90% conversion rate lift from a single idea or test. But those are aberrations and, if the result of a highly disruptive test, were earned with great risk associated, no doubt. Digital optimization is about the accumulation of data, in the interest of making decisions that are most likely to “connect” and less likely to “strike out.” Pure and simple. This applies to every aspect of digital business – marketing, communication, design, experience, product development, engineering, etc. Every decision, every expense, every resource allocated in these areas constitutes an at bat that can either lead to more productive future at bats and eventual value (runs) or a wasted opportunity (outs).

If you want to employ Moneyball-thinking in your digital business, make sure that you are capturing and testing every distinct hypotheses constantly. Be honest with yourself and your team about how you approach your at bats. Do you take big swings based on gut and habit? Or do you quickly and continuously iterate and learn, accumulating data so that your “big swings” are more likely to connect based on the data earned? The transition from the former to the latter is not easy. But it’s very much possible. In fact, that’s in no small part why Clearhead exists.

Companies that think digital optimization can be pursued as a series of forced, big swings at the plate in search of home runs should know that this approach comes with great risk of negative performance impact and great expense. Without doubt, there will be occasional “winners,” but they will inevitably be burdened by the cost of risk and waste. When do we believe in swinging for the fences? When you have a lot of data about what works – when you know the pitcher’s strengths and weaknesses and understand the context and are in a hitter’s count. You arrive here with the benefit of many at bats and a lot of data to inform your guess.

All of which takes me to a prevailing concept for the success of a digital optimization program. While it is common to chase and glorify the big testing and conversion optimization wins, a digital optimization program’s value should be equally correlated to waste and “outs” avoided. Everybody wants winners. But if not fully integrated into the design and build philosophy and rationalized against real cost and opportunity cost, analytics and testing can devolve into dabbling that does not drive the big bets. When properly designed, you will see a team of smarter hitters across the board. A team informed by a wealth of data. A team able to get on base with increased frequency. And a team less likely to make big outs in crucial situations. A team more likely to score through walks, singles and doubles. And, occasionally they will hit one out of the park. And they will be able to describe exactly why they swung for the fences that time.