November 17, 2015

Real Talk: Investing in a new optimization program

You asked, we answered. In this new series, we get real and answer some of the most common questions we hear in the market and from our clients. Is there something you want to know about optimization, testing and personalization? Email your questions to

What are the initial investments I need to launch an optimization program?

If you’ve been listening to the market and your peers, you have probably heard that optimization has transitioned from an added bonus to a necessary capability for future digital growth. Maybe you’ve been lobbying for an optimization program and it’s finally time to put together a budget for the upcoming year. Or perhaps you’ve been pitched by a software company and are thinking about the total cost of ownership. But even if a SaaS procurement cycle instigated this process, you need to think about people and process before you buy.

The software required to do testing, targeting and personalization is your dial tone. It’s what enables you to target certain audiences and compare their behavior. But on its own, software doesn’t get testing off the ground — the tool does not equal the program.

An optimization program is all about the people, processes and resources you apply, to the testing software as well as the related suite of analytic and customer experience capabilities. You must take all these facets into consideration when looking at the initial investment needed to succeed at testing, optimization and personalization.

What does the team look like?  

The first step in establishing a program is assigning someone to manage optimization. You could have multiple people in this role, but at the very least it is an entire dedicated resource, not a fraction of someone’s job. It is not enough to have a product manager or analyst running a testing program on the side. If optimization is going to be in the DNA of your digital experiences and products, this should be a dedicated function with the internal capital needed to tap into multiple departments.

The optimization lead doesn’t necessarily need to be a product manager or analyst. Our bias is that this person should understand the relationship between product, UX and data, analytically as well as financially. When faced with limited time and resources, they can optimize a profitable outcome for the company that also improves the customer experience.

In an ideal scenario, this person would also have a team to help extract data, prioritize hypotheses, create tests and ensure the results fuel continuous improvement. Contrary to popular belief, tests are not commonly designed and executed with drag-and-drop editors, and truly rigorous analytics requires going beyond the basic stats. This typically would require skilled resources for:

  • UX design to propose solutions and create tests
  • Front-end development to build tests, QA and push results to analysts
  • Analytics to calculate data requirements, find key segments and measure results

Ideally you’d have dedicated people for each function. Additionally, every single product manager should be tapping into optimization, and vice versa. But at the very least you should have team members from each of these departments who are invested who are invested in optimization. Even in an MVP scenario, everyone involved should have goals tied to the program’s success.

What support will the team need?

The best team still needs a process to succeed. Without a process, you are simply testing things randomly — by gut, based on an article someone read or dictated by internal politics. This often leads to nibbling at the edges of conversion rate while the rest of your investments go unvalidated.

Optimization programs need a process for using data to identify problems that your business or customers want solved, as well as prioritizing which problems are worth solving in a given period. You need a process for leveraging data like customer feedback, user testing, previous experiments and analytics to develop, prioritize and sequence hypotheses. You need a process for developing, QAing and launching experiments. You need a process for monitoring the data as it comes in to ensure that you know whether something is helping or hurting the business. Once you have a sufficient amount of data, you need a process for going back and determining key metrics and supporting metrics. And finally you need a process for sharing and acting on these metrics.

If you can do all of this in a way that is self reflective, over and over again, you have an optimization process. Your next challenge is will be getting your people and investments aligned with this process.

Getting the team on board probably includes one-on-one training. Every stakeholder should understand what testing is, what personalization is, the key statistical terms they’ll  be working with and the benefits of this new process over the previous approach. You then need to have a series of handshake meetings between the optimization organization and those departments most at risk for challenging it. Moving forward, you also want to carve out time for regular business review, to not only see what tests were successful, but also to see how adoption is progressing and to help mitigate any risks.

It is worth acknowledging that some of these roles, expertise and approaches are new. They are emerging capabilities that are hard to find in the market. While you’re onboarding a team and process, you may want to leverage agency resources to help fill gaps and accelerate adoption. Or you may find that an ongoing engagement with a third party is the best way to scale up and across your organization. Beyond manpower, a proven process is the most valuable thing an agency can bring to your program.  

What software will I need?

SaaS optimization tools give you the ability to determine which users are shown which experience, then compare the results of those experiences to different user sets. Today’s optimization platforms include pretty much everything you need to run AB test, multivariate tests and personalization. So if all these tools solve for basic blocking and tackling, how do you decide which one is right for you?

There are six questions you need to consider before signing your SaaS contract:

  • What is the cost to license?
  • What is the overall cost of ownership?
  • Do you trust the team you are buying from to support your unique needs?
  • Do you believe their product roadmap will solve innovation problems?
  • Do they have unique features or functionality specific to your industry?
  • Do they have a data philosophy that is uniquely beneficial to your business?

For many, the decision is heavily influenced by budget. Assign a certain value to the program to determine what you can spend, and don’t forget to include implementation and continuing support in that mix. Some platforms are more cumbersome and difficult to implement, though they have a broader set of capabilities. Bells and whistles almost always indicate a higher cost of ownership, but they may be critical for your business.

You should also consider team behind the tool. Who will be supporting you?  What is their product roadmap? You are betting not just on what you are buying today, but on the platform’s ability to deliver what you need 12 months down the line.

And yes, there are functionality and philosophical differences between the tools. If they cater to a specific industry or use case that fits your company, that can be a huge advantage. Related to that is how they deal with data, whether they build upon multiple data sources or build their own data layer.

The alternative to purchasing SaaS is creating a proprietary tool that you would engineer and build yourself. The cost to do that and support that is significant, and the product you build would likely be inferior to what’s now available in the market. You may also have a basic modular AB testing capabilities in your CMS, though these extension are limited in what you can change, the rigor of statistics and measurement is much more limited, as well as the ability to personalize and target visitors. With the exception of a few edge cases, licensing a SaaS platform is the only way to go.

By purchasing the right tool for your business while also investing in people and process, you will have established the foundation for an impactful optimization program.

  • Austin

    3601 S. Congress Ave.
    Bldg. C302
    Austin, TX 78704
    United States
    View on Map
  • Cincinnati

    1311 Vine St
    Cincinnati, OH 45202
    United States
    View on Map
  • London

    14 Grays Inn Road
    WC1X 8HN
    United Kingdom
    View on Map
Map of Clearhead’s location in Austin, TX

Let’s talk.