June 6, 2017

Three reasons to combine A/B and UX testing

Combining UX Testing and AB Testing

Understanding how people use your product is very important if you want to achieve customer satisfaction with your product – and keep them coming back. But how do you decide when to use A/B testing, UX testing or both?

UX testing is commonly thought of as an additional step that slows down the velocity and increases the cost of a project. If we’re going to A/B test a new feature or design and see what does better, why do UX testing at all?

At Clearhead, we’ve seen three ways you, your business and your customers will benefit from pairing A/B testing with UX testing:

Reason #1: Understand “why” a test won or lost

After an A/B test, you get a robust set of analytics data which shows “what” happened.

  • What did customers do on the site?
  • Did customers convert to order more often?
  • Were they more likely to sign up for an email list?

What you can’t get from the A/B test is the “why.”

  • Why did customers not use a new feature?
  • If you roll out a design on desktop and mobile, why did it improve orders only on mobile?
  • Why did customers search more often?

Your storytelling on an A/B test can only go so far based on analytics and UX expertise.

The qualitative data from a UX study can answer why customers and users do what they do. By both A/B and UX testing a design or feature at the same time (or after an unexpected A/B testing result), you can pair the “what” and the “why” to have a full understanding of the test results. Complementary quantitative data from A/B testing and qualitative data from UX testing provides a more complete data set for better decision making.

Additionally, while an A/B test may give some signal of what to test next, a UX study can give clear direction directly from your customers on what to do or test next.

Reason #2: Reduce A/B testing risk by putting your best designs forward

You can catch usability problems before running an A/B test. Performing UX testing before launching an A/B test allows you to:

  • Inform the designs being A/B tested and assure you’re putting your best foot forward in an A/B test.
  • Be confident that an A/B test didn’t lose due to usability problems.
  • Uncover problems with your new feature or design before it goes head-to-head with the current experience in an A/B test.

A/B testing puts real experiences out there – and if it’s bogged down with usability problems, testing can hurt conversion.

You can reduce risk, since UX tests don’t impact customers. When UX testing comes first, there’s no risk of negatively impacting customers. Additionally, you can be more confident about what you’re putting forth in an A/B test and continue to make iterations that are user-friendly and drive conversion.

Most importantly, you’ll reduce the number of A/B test iterations you’ll need to do, save money and be able to move on to other big problems.

Reason #3: Discover external customer problems to inform solutions

Using UX testing (and other tools in Problem Solution Mapping), you can observe customer friction points and understand their impact. This allows you to prioritize and solve problems that will have the biggest impact.

By hearing directly from customers, you can reduce the bias of being too close to your work and blind to problems your customers face.

UX testing gives you the chance to see a product through the eyes of your users – essentially “walking in their shoes” – to uncover the problems standing in their way to converting or using your product to achieve their goals.

Watching your customers interact with the product is also one of the best ways to generate hypotheses.

Bonus: Knowing your customers’ goals helps build better products

Beyond pairing UX and A/B testing, UX research methods can be used to learn more about your customers. You can learn how customers feel about your brand, how they are using your products and what problems they have (that you may be able to solve through a new feature or product). You can use formative research methods to learn about your customers and make sure your product roadmap matches up to their problems and needs.

By combining A/B and UX testing, you’ll produce far better and faster results for your business and customers. Your customers recognize – and expect – good design, and one of the best ways to improve design is to understand what customers are doing on your site and why.

Looking for help embarking on your own Optimization journey? We’re here for you.