9 product experimentation tools for better A/B testing

In this chapter of our product experimentation guide, we’ll take you through nine different experimentation tools to add to your product stack to diagnose, ideate, test, and evaluate optimizations.

What A/B testing tools won’t show you

On their own, A/B testing tools only show you which variation is winning, not why.

To design better experiments and impactful feature updates, you need to understand the deeper motivations behind user behavior and avoid ineffective solutions like guesswork, chance, or the opinions of large semi-aquatic mammals (aka HiPPOs—highest paid person’s opinion). 

Another drawback of A/B testing is that it can take time to generate statistically significant results (especially when you have low traffic). And when you're part of an agile product team and need to make iterative changes, that’s time you don’t have—you’re likely eager to understand what’s working (and not) straight away.

So what’s the ideal solution? Instead of focusing on A/B test data in a vacuum, you can combine test results with qualitative insights (the kind you get from session recordings, surveys, and feedback tools); this ‘best of both worlds’ approach allows you and your team to optimize your product for user experience and business goals.

2 free product experimentation tools to get started with

Product experimentation doesn’t have to be expensive: small teams, startups, and even complete beginners can collect user insights and A/B test variations for free by using these two popular tools together. 

1. Google Optimize

Google Optimize is a free website experiment platform that lets you run A/B, multivariate, redirect, personalization, and banner split tests to compare their performance and improve goal conversion. Anyone with a Google account can sign up, and Optimize integrates with Google Analytics, Google Ads, and select third-party insights tools (including Hotjar—more on this in a second), making it easy to set up and combine metrics from multiple sources and add qualitative user data to your experiments. 

The optional Google Optimize Chrome extension also provides a visual editor for creating test variations and previewing experiments without coding knowledge.

2. Hotjar

Hotjar (hi there!) is product experience insights software with a free plan for up to 1,050 sessions/month and four main tools: 

• Heatmaps: to see where users click, scroll, and move their mouse

• Recordings: to view how users browse across an entire session

• Surveys: to ask users direct questions

• Feedback: to let users tag, rate, and comment on any page element

Hotjar is designed to give you actionable product insights so you can spot common areas of friction in the user experience, get ideas for improvements, and measure if your fixes are working (and why).  

While Hotjar is not an A/B testing tool, it’s integrated with some big names like Google Optimize and Optimizely, and works with most usability testing software by using JavaScript Event targeting to filter insights by test variation. 

3 popular A/B and multivariate testing tools to set up experiments

Split testing is still a big part of the product experimentation cycle, and you’ll need the right software to get it done: here are three popular A/B testing tools that let you create, measure, and report on experiments.

1. Optimizely

Optimizely is a digital experience platform with multiple tools, including web, server-side, email, and B2B commerce experimentation. With Optimizely’s popular Web Experimentation tool (previously known as ‘Optimizely X’), you can run A/B and multivariate tests and personalization campaigns to improve conversion rates and user experience. 

2. Omniconvert

Omniconvert is an experimentation tool used by product and ecommerce teams to create A/B tests, personalization experiments, pop-ups and overlays, and trigger on-site surveys.

Omniconvert’s main A/B testing tool is called ‘Explore’ and offers advanced segmentation to create experiments for specific user cohorts. You can segment by over 40 parameters, including geolocation, on-site behavior, and traffic source, to run experiments that only target a specific audience. 

3. VWO

VWO is a conversion optimization platform for creating A/B, multivariate, and split tests. VWO’s main experimentation tool is called VWO Testing and is intended for marketing and conversion rate optimization teams to create test variations without coding using a visual editor. 

Another tool, VWO FullStack, is designed for product teams to run server-side A/B tests without impacting performance. VWO FullStack also has a feature rollout tool, allowing you to release new features to small groups and measure user impact before launching it to all customers. 

4 tools to get test ideas and see why your A/B tests are winning

Split testing starts with solid hypotheses founded on good insight. These four tools will help you collect foundational insights and understand why winning tests succeed, so you can double down on what’s working and create a virtuous cycle of data-backed optimization.

 1. Heatmaps

Heatmaps give teams an at-a-glance overview of what users are doing in a product, making them a great starting point for developing testing hypotheses. 

Once you’ve run an experiment, heatmaps will give you a deeper insight into why one variation outperformed another. For example, using Hotjar Heatmaps, you can monitor A/B tests on all variations and compare results to get a clear, visual overview of how click and scroll activity differed. Plus, you get to showcase results to team members and stakeholders—it’s hard to argue with a heatmap!

 2. Recordings

Session recordings (aka user replays) are renderings of the real actions an individual visitor takes as they go through a session on a product or website, from entry to exit. Since they measure all visitor actions, recordings are great for finding where people hesitate or get stuck, giving you ideas for fixes and improvements to test. 

Recordings can also help you understand why some split testing experiments don’t result in a winner, and give you qualitative insight from low-traffic tests that may not have a statistically significant sample size. 

If you’re using Hotjar Recordings, you can use Event filters to view session replays from any A/B test variant and see what users did throughout their session, and how their collective behavior affected your test results.

3. Surveys

Surveys are a quick and easy way for teams to collect feedback directly from users. Knowing what real users actually think about and need from your product helps you build experiments around user experience and reveals issues and opportunities that might come as a surprise to your team. 

If you’re using Hotjar Surveys, you can set up Events targeting to trigger surveys within A/B test variations and collect valuable qualitative data during experiments. Using A/B test surveys alongside quantitative data will give you a fuller picture of not just what users did (e.g. where they clicked), but what they thought while doing it (e.g. do they understand what a new feature does?, do they like the new color scheme?, is there something they’re missing?).

4. Feedback

User feedback is any information collected from customers about their experience with a product or service. Feedback can be quantitative, i.e. numerical, like a Customer Satisfaction (CSAT) score, or qualitative, like an open-ended question response. 

Hotjar’s Feedback tool collects both quantitative and qualitative feedback: users can highlight any page element, rate it, and provide additional comments if desired. You can use Hotjar Feedback in A/B test variations without interrupting the user journey with a pop-up, giving you more context behind how users feel about design changes or new features.