April 14, 2017

Vinyl and Server Side Testing: What Once was Old is New Again

There's a lot to consider when deciding whether to test server side or client side.

The cyclical nature of trends can be seen all around us: beards, Polaroid cameras, Mark Wahlberg. But perhaps the one that holds the most meaning for Clearhead is the cyclical nature of music.

In 1988 the popularity of vinyl slowly began to decline as CDs and later MP3s gained in both popularity and market share. But in 2016 this came full circle as vinyl record sales reached a 28-year high.

Insound, one of the very first online music stores, was a 1990s start-up co-founded by Clearhead founder, Matty Wishnow. The little engine that could, Insound, was both underfunded and margin restricted. After sinking a significant amount of both time and money into designing and building a product that ultimately barely made an impact  (and nearly sinking the business), the Insound team quickly realized that shots in the dark were no way to run a business.

A stats nerd at heart, Matty soon became obsessed with creating a culture of experimentation–not only out of curiosity but also out of necessity. Test and learn then became the norm for Insound, guiding every business and product decision they went on to make. This marked a huge positive shift in both philosophy and revenue for the business.

Testing Like It’s 1999

Operating in a price sensitive environment, Insound’s margins were thin. Changes that resulted in even a single point increase in margin had a significant impact on the business. Thus, confident changes via testing and iteration proved crucial to its success.

Insound found that tests involving search results, category sorts and pricing were often big business movers, as it allowed them to better understand the price elasticity for products as well as their customers’ sensitivity to price.

Insound built these experiments–from the search algorithms to the pricing rules to the sorting rule–for these tests entirely from scratch and server side since there was not yet software to help with the process (think late 90’s). This was an extremely resource and time intensive process, as you ostensibly had to deploy multiple instances of code on multiple servers, while using a proxy server to route users to different experiences.

Even the analysis was laborious: pull and parse the log files, and then hand calculate the statistics on them. But the impact the tests had on Insound’s business made all of the effort worth it.

Clearheaded and Fast Handed

Fast Forward to 2013. Clearhead was just a year old when Optimizely was born. Some of the first tests we ever ran together were with Insound, and the first real winner was a hybrid client side/server side test where we ran another version of that same successful search algorithm test.

Using server side testing to optimize Insound's search algorithm
Clearhead used a hybrid server side/client side experiment to optimize Insound’s search algorithm.

We launched an experiment to help us determine which factors should most heavily influence the search results logic for the Insound website (if you follow the Optimizely blog this may sound familiar). While many at the time thought of Optimizely as a dead-simple way to manipulate front-end design changes in experiments, the test Clearhead planned used Optimizely as a “quarterback” of sorts.

We defined four permutations of the search results page, based on different factors weighing the algorithm. Optimizely ran command and control to manage which traffic was seeing which search results, helped us with targeting (excluding certain behaviors and mobile traffic) and, of course, helped us measure for significance.

This test, an ahead-of-its-time hybrid of both server side and client side testing, was a big winner for Insound, with the new search algorithm beating out the control’s conversion rate by 39%. Client side testing and Optimizely made it possible to do all of this–which used to take many weeks–in less than a single week.

Clearhead optimized Insound’s search algorithm using a hybrid server side/client side testing approach.

What’s Old is New

Fast forward three more years to 2016, and to the release of Optimizely’s Full Stack product. We now have the ability to do deep client side and server side testing–far beyond the tests that most are even running today–in a much easier manner than ever before.

To be clear: server side testing, while it’s new for many people, is not really a new way of doing experimentation. Despite the resurgence in popularity of records, vinyl never went away–it was surpassed by something more convenient but not necessarily higher fidelity. The same goes for server side tests.

Companies have long been conducting server side experiments. The difference is that they have, by and large, not been leveraging third-party software to do these tests. What Optimizely has done, in the same way it democratized client side testing, is democratize server side testing.

A Time and a Place: When to Test Server Side vs. Client Side

Business operators should now think about experimentation as a spectrum–not a binary decision between client side and server side tests.

What will be the most effective and efficient test method for your team to run for each particular test? That may mean choosing a client side test, a server side test or even a hybrid approach.

There are no hard and fast rules for when to choose a server side test, but there are some questions to ask to point you in the right direction:

  • Can the test even be executed client side? (in most cases, the answer to this question is “yes”)
  • What’s the time and effort of developing the MVP experiment client side vs. server side?
  • How easy will it be to adopt this test into my code base if it’s a winner?
  • How backlogged is the engineering team?
  • When is the next code deployment?
  • How will performance be impacted by this test if we deliver it client side?
  • How difficult will this be to code with a front-end approach? Will my team have to build complicated workarounds to make this test communicate with other systems?

The capabilities of traditional client side testing are astounding, but for more complicated and dynamic experiments, server side testing via Optimizely Full Stack may be the answer. This new product opens up a depth of options and a power of experimentation that complements client side testing like the vinyl record compliments your Spotify account.

Back to the Basics: PSM

From load and performance to sort and search to data validation to pricing, the opportunities for experimentation with the introduction of Optimizely Full Stack are seemingly endless. With the ability to test deeper in your marketing stack more easily than ever, you may find yourself asking: Where do I start?  

This is where a solid strategy becomes critical. Start with the basics (if you’re a Clearhead client you’ll know that means with PSM). Clearly determine the most important problems you’re trying to solve, develop and prioritize your hypotheses for overcoming these challenges and–only then–examine whether a front end, back end or hybrid test would be the best approach to get you there.

Clearhead's problem solution mapping methodology
Clearhead’s PSM framework helps you determine which experiments are most likely to result in ROI.

Always be sure to test solutions that may solve your biggest problems, thus representing the greatest odds of making a big impact on the financial outcomes for your business (what we refer to as The Physics of ROI).

As with vinyl and testing, what was old is new again. If you’re not sure where to start, or if you’re looking to take server side testing for a spin, let’s talk.

  • Austin

    3601 S. Congress Ave.
    Bldg. C302
    Austin, TX 78704
    United States
    View on Map
  • Cincinnati

    1311 Vine St
    Cincinnati, OH 45202
    United States
    View on Map
  • London

    14 Grays Inn Road
    London
    WC1X 8HN
    United Kingdom
    View on Map
Map of Clearhead’s location in Austin, TX

Let’s talk.