10 Qualitative and Quantitative Sources to Feed Data-Driven A/B Testing Hypothesis

This article is for anyone interested in improving their A/B testing skills.

In today's online business world, optimising the performance of your website is not a luxury, it is a necessity. A/B testing is a powerful tool that allows you to experiment with different versions of your web pages to see what works best for your users. But where do these A/B test ideas come from, and how do you ensure they are data-driven and effective?

Below we've compiled a list of 10 sources to help you in create data-driven A/B testing hypotheses. Each source has its own focus and therefore needs to be considered separately. We have categorised them into qualitative and quantitative research to highlight their different contributions. Insights from qualitative analysis guide us in asking the right questions and formulating targeted hypotheses, while the quantitative data lend credibility to these hypotheses with measurable outcomes.

By asking targeted questions across qualitative and quantitative data sources, you do not only gain valuable insights, but also a deeper understanding of how your tool works. The combination of qualitative and quantitative approaches provides a comprehensive analytical foundation, strengthening your decision-making and maximising the success of your hypotheses. Now, let's look at how each source can benefit you and guide your approach to data-driven hypothesis development.


Qualitative Sources for Data-Driven A/B Testing Hypothesis

1 Session Recording

Session Recording provides a first-hand view of user interactions, offering valuable insights into navigation patterns, user engagement, and potential areas for enhancing the overall user experience.

  • Which interactions during the session cause user frustration?
  • Are there particular forms that could lead to confusion or abandonment?
  • Which areas of the site are frequently overlooked by users?

2 User Feedback

To understand user sentiments, preferences and concerns user feedback through surveys, reviews and social media serves as a direct channel to gain a comprehensive understanding of the user experience landscape.

  • Which problems do users frequently mention?
  • Which features or improvements are most often requested by users?
  • Which aspects of the site generate positive feedback?

3 User Testing

User Testing helps analysing workflows, measuring response to design changes and identifying sources of confusion to improve the user experience through real-time insights and preferences.

  • Which workflows do users find difficult?
  • How do users react to specific design changes or new features?
  • Which content is causing confusion?

4 Expert Reviews

A critical evaluation of UX, conversion optimisation, or web design offered by expert reviews provides actionable insights to enhance overall user experience and website performance.

  • Put on the user’s hat and understand the flow.
  • Which areas might be causing confusion?
  • How to generate user trust on the site?
  • How to reduce user anxiety?

5 Competitive Analysis

Industry benchmarks, market positioning, and opportunities for strategic differentiation that guide informed decision-making for a competitive edge can be revealed by a competitive analysis

  • Which are the industry trends/best practices?
  • Which compelling features have our competitors implemented that we lack?
  • Which personalised experiences are competitors offering to increase user engagement?

6 Customer Support Data

In order to gain insight into user issues, preferences and sentiments, customer support data that helps achieving increased customer satisfaction is necessary.

  • What are the most common support requests?
  • Which feedback do we get from users?
  • How can we improve processes on the site that often cause confusion?

Quantitative Sources for Data-Driven A/B Testing Hypothesis

7 Heat Maps

Visual insights to quickly identify trends, patterns and outliers are offered by heatmaps that at the same time provide a visually intuitive perspective on user behaviour.

  • Which areas of the site receive the most attention from users?
  • Are there certain elements that users interact with more often?
  • Are there areas which might be overseen by users?

8 Web Analytics

Web Analytics empowers a comprehensive understanding of online user behaviour and enables data-driven insights for informed decision-making and continuous website optimisation.

  • Which pages have the highest bounce/ exit rate?
  • Which pages have the least interaction?
  • Which channels are driving the most qualified visitors?

9 Conversion Funnel Analysis

An analysis of the conversion funnel streamlines the customer journey by pinpointing bottlenecks through conversion funnel analysis, enhancing overall efficiency and optimising the path to conversion.

  • Where in the conversion funnel are the highest drop-off rates?
  • Which funnel steps could be optimised to increase conversion rates?
  • Are there unnecessary obstacles in the conversion funnel that could be avoided?

10 Lessons Learned from Previous A/B Testing

Insights gained from past A/B testing provide valuable lessons that guide informed decisions for future optimisation strategies.

  • Are there tests that have had no significant impact but show interesting trends in segmentation?
  • Which insights worked best and could be extended to other pages?
  • Which patterns can be identified from looser tests to better understand the user behaviour?

In the ever-evolving landscape of data-driven decision making, generating effective test ideas to improve your online presence is both an art and a science. Never forget that the real power is in the synergy. The most effective optimisation strategies come from combining quantitative and qualitative insights. But how do we turn data-driven ideas into powerful A/B testing hypotheses? The answer lies in our next article “Crafting Powerful Hypotheses: Unleashing the Potential of the PSR Framework”, coming soon.

Vanessa Mangano

Vanessa Mangano is a Consultant at Up Reply, specializing in personalisation and experimentation. Her responsibilities include expert consulting on Optimizely for various international markets. In the blog, Vanessa will share tips and tricks for Optimizely, along with interesting insights from her hands-on experience.

Let’s Take Your E-commerce to the Next Level

Unlock new opportunities and redefine the customer experience through personalised, data-driven strategies with Up Reply.