This marks the second post in our series on things we hear while selling testing and optimization services.
This edition finds us in the pitch with a senior digital business director at a large travel oriented business. She has a year full of plans that include re-platforming their email system and overhauling their mobile experience.
The conversation continues as we explain how our services can help her make data driven decisions on features and designs she may want to use in her new mobile experience. “That sounds great,” she says.
At this point we believe we are on the verge of a new and very exciting partnership with a company we know we can help, when the conversation suddenly turns to “timing.”
“You know.” she says. “I am excited about the possibilities here, but I want to be honest about the timing. Like I mentioned, I have a very full roadmap planned for this year. Testing is on it, but not until the back half. Let’s talk again in 6 months.”
“Well, we can certainly appreciate that,” we say. “But have you considered that testing and optimization should be the driving force that refines and prioritizes your roadmap and not an item on it?”
“Okay, thank you very much for your time. Please keep us in mind when you get to the testing portion of your roadmap.”
And since that moment, we’ve heard similar comments from no less than three other senior digital leaders from equally large businesses.
So what’s going on here?
I suppose this is not surprising. In a world where every day there’s a hot new tech start-up you should meet, a fancy new feature being launched by a competitor, and an endless supply of “table stakes” that are “no brainers” to work on, the idea of taking a minute to validate your assumptions before moving forward can feel, well, slow.
With that said, how many times have you done the deal with the hot new start-up or finally rolled out that “table stakes,” “no brainer” of a feature, only to find it didn’t materially move your business forward?
And what if I told you there’s a way that you could get a data driven read on these decisions before you fully invest in them? That’s what a data driven, “hypothesis validation” program can do for you.
Here’s a real life example. One of our fantastic software-as-a-service, subscription based clients had the idea that giving away a free, tangible gift in exchange for signing up for their 30-day free trial would lead to explosive growth in trial sign-ups.
They were so confident in the concept that they were ready to move forward with a partnership with a shiny new start-up that actually helps SaaS companies manage free giveaway programs. “This is a no brainer,” they said.
The great news for us was they are an existing client with whom we had had success debunking the value of a previous “no brainer” via the hypothesis validation program we run for them. They are now believers in using testing to validate their roadmap, so when we suggested a very simple, minimum viable test wherein we’d hardcode an offer for a free giveaway to 50% of their leads and then manually fulfill them via Amazon to see if over the course of three days there was any sign this giveaway offer would lead to the explosive trial growth the expected, they jumped at it.
You can probably guess what’s coming next. After three days of offering the free giveaway, the results were completely flat.
Now the learning for this was not that their leads don’t like free stuff, but rather, that they weren’t primed enough from their landing pages and value prop presentation to be receptive the free offer. Kind of like when you walk by a booth for a company that you’ve never heard of and the first thing you are greeted with is “free T-Shirt”.
And the best part – landing page and value prop updates were on the roadmap for later in the year and after the giveaway program rollout. So this very simple experiment drove a data driven change to their roadmap priorities and increased the odds that their roadmap will actually drive real value. That’s what roadmaps should do, right?
How can I make data driven, product roadmap decisions? Don’t worry — here’s three steps to get you started.
1. Identify the assumptions in your roadmap
Whether you have a formal roadmapping process or just a list of stuff you plan to work on this year, you have a roadmap. In it lies your assumptions on the things you think will improve your user experience and help your digital business. Identify them and write them down. Even if you think it’s “table stakes”, write down why you think so.
A 2013 table stake example: “A responsive mobile site.”
A likely assumption: I believe if we build a single website that automatically adjusts it’s layout and features based on the devices it runs on, we will sell more products on phones and tablets.
2. Test those assumptions as simply as possible
Before you greenlight any major project, go through each assumption you’ve written down and identify the simplest way you can come up with to confirm whether there’s any merit to it.
An example of a minimum viable way to test the responsive site assumption above:
Take one page type, say a product detail page if you run an ecommerce store, and make that responsive to the device you believe is most important, say the iPad. Now test this one page on iPads. Does it convert materially better than what you offered before?
If so, great. You know you are headed in the right direction. If not, you need to adjust your assumptions… and the approach to your roadmap.
3. Evolve your roadmap
Each test you run exposes a card in the proverbial poker game of your business, and, in turn, provides valuable information enabling you to make bets with better odds.
Your roadmap should continuously evolve as you obtain data through minimally viable tests and make better informed bets.
We recommend that Testing & Optimization be the core process — the “table stake” — that defines and continually refines your roadmap. The key benefit being that your major decisions are more likely to deliver material revenue growth for the business than those based on gut, a vendor’s fancy sales pitch, or an educated guess.