What comes to mind when you think about experiments? Is it a professor in a lab playing with test tubes and Petri dishes? Maybe it’s creating potions as a kid with the contents of your parents’ bathroom cabinet? Or is it product teams testing feature changes to find the best solution?
All of these answers are valid, but there’s only one we’ll discuss today. Experiments are essential for optimizing your product. They enable you to test new feature designs and ideas to determine their effectiveness.
Experimentation is a key part of the product development process that shouldn’t be skipped (however tempting jumping ahead to product-wide deployment may be).
So, put down your Petri dish and step away from that cabinet—it’s time to talk product experiments (lab coats optional).
Product experimentation is a crucial part of your product development process.
In-product experiments help you test assumptions to validate or reject product development ideas.
All product changes and updates should be based on user needs and testing insights, not guesswork.
A/B testing, multivariate testing, click tracking, usability testing, and more can be used to conduct in-product experiments.
What are in-product experiments? #
In-product experiments involve continually testing hypotheses in order to validate or reject product development ideas and initiatives. They enable product teams to improve products and features—and ultimately the user experience—with real insights and data on what users want.
User data and insights are an integral part of in-product experiments. Your initial hypothesis to test should aim to solve a problem you know users have.
This shouldn’t involve taking a shot in the dark—you should identify real problems through user and product research. Asking questions, testing ideas, and validating initiatives are key for continuous product improvement.
There are often multiple ways to solve a user problem, but you’re always aiming to find the best solution.
What is continuous discovery? #
Continuous product discovery is the process of consistently researching user needs and UX issues in order to identify problems and experiment with ways to solve them. It is key for making informed product decisions.
Research involves quantitative methods as well as constantly communicating with users—whether that be via in-product surveys, user interviews, or any other form of user research—in order to find opportunities for improvement, validate ideas, and prioritize solutions.
This is essential for ensuring your product continues to meet your user needs, while also improving the product for future users.
How to build a culture of product experimentation #
Effective product experimentation requires you to bake the process into everything your product team does. It involves establishing a culture of product experimentation in your organization. Here’s how.
Make experimentation a habit for your team #
Constantly researching, hypothesizing, testing, and iterating is an approach to product development that not all companies use. They should—but they don’t.
It might take some effort to ensure the entire team understands the importance of data-backed development—but it’s well worth it. Plus, it doesn’t just stop at your product team’s day-to-day workflows.
An experimentation mindset is a strong tool on any team, whether it’s customer service, sales, operations, or any other department. Testing new approaches to the same processes will help your team improve the user experience.
Continuously identify opportunities for optimization #
Test, test, and test again. That’s how you build a culture of product experimentation: with ample practice.
Continuously collect data insights on user behavior from product analytics, user feedback, and in-product investigations.
Data analysis of these insights can help you identify issues to be solved and spot opportunities for improvement.
Establish processes and infrastructure you will use #
A product experimentation framework is a must for building a product experimentation mindset amongst your team and company.
Make sure everyone knows how experiments are conducted in your company—both in terms of the process, the tools used, and the metrics for success.
Following a structured, documented process ensures that all team members, including new hires, are able to get involved. It could be a good idea to create a handbook on your approach to product development that outlines the experimentation method and tools.
The components of a great product experiment #
So, what makes a great product experiment?
Put simply, it’s the same thing that makes any experiment great: the scientific method.
You likely learned about this in your middle school science classes, but it applies here too. The scientific method is all about observation, measurement, and experimentation.
Like most things when it comes to improving products, experiments start with user research.
Conduct user research #
User research helps you identify the friction users are encountering, as well as spot low-performing features or pages. Here are a couple of ways you can test the effectiveness of your current designs:
Microsurveys: Ask users about their experience directly after performing the action you’re asking them about
Click testing: Run click tests to see where users expect to find things on a webpage
User interviews: Chat with users about their experience in their product. Do they like the interface? Can they navigate through your site easily?
This helps you hypothesize potential solutions to the problems outlined by users without it being a guessing game.
For the sake of making the explanations easy to follow, we'll include the following example throughout.
Let’s say your user research found that 82% of users exit the onboarding flow when asked to integrate other tools during step three of four. They click out of the product walkthrough within 2-3 seconds of the integration CTA showing up. In this case, you’re looking to increase onboarding flow completion—the percentage of users that reach step 4/4.
Create a clear hypothesis #
Research could unearth a wide variety of potential optimizations—but you can’t activate them all at once. Prioritize user issues based on impact and user needs in order to be clear about what you’re testing.
However, hypotheses are merely assumptions. You’ve identified the problem, and now you’re speculating about how you could fix it. Testing is where you validate or reject this speculation.
This distinction is key for product development: hypotheses are not facts. Doing user research isn’t enough to fix problems—what you consider a fix, clients may not consider adequate.
In the example we set above, your hypothesis could be: “Moving the integration CTA to the final step of the onboarding flow will increase onboarding completion.” See how we’re only talking about making one change? And there’s no mention of increasing tool integration. You want to focus on one thing at a time.
Define what success looks like #
Here’s where you clarify what constitutes a successful experiment.
Start by asking these questions:
What metrics validate your new experiment?
What are you considering a successful result?
In our example, product experiment success could be defined as an X% increase in onboarding completion. It could also be an X% reduction in onboarding drop-offs at step three. Keep in mind: It’s essential you know what validates a hypothesis before testing it.
Assign your team roles #
Product experimentation requires collaboration—you want to quickly create effective experiments, and that requires collaboration.
Who’s in charge of designing the experiment?
Who’s in charge of implementing it?
Who will monitor the data to gather insights?
You also need to ascertain what tools you’ll need for the testing process. Where will you collect data? How will you gather continuous research? Make sure you select tools that can help you run and track product experiment tasks and have complete control over your product adoption.
In our example from above, you need someone to manage the project, design the changes, create prototypes for testing, implement the changes for testing, and analyze results. One person could do more than one task here, so we’re looking at around 2-3 team members for a speedy product experiment.
Monitor results and iterate #
You won’t always effectively validate your hypothesis with one round of experiments. Sometimes you need to test over and over again until you get to the right solution. And, even when you implement changes, you’d want to keep testing to ensure they remain effective.
The product experimentation process never ends—it just restarts, albeit sometimes with different experimentation methods. It helps you make better decisions and provides valuable insights to support product growth.
🦎 Using a tool like Chameleon and integrating it with your tech stack is a great way to quickly and easily run and monitor product experiments. You can use Tours, Tooltips, Microsurveys, and Launchers to present new flows and features—and give users a better product experience.
Leverage Chameleon’s two-way integrations
You can integrate Chameleon with Mixpanel, Segment, Heap, Freshpaint, and other tools. Connect your tech stack, start tracking events and the performance of your in-product experiences, and get a complete overview of user behavior.
6 types of in-product experiments to make better decisions #
In-product experiments come in all shapes and sizes, and deciding which is best for your goals is a must. It depends on the feature in question, the stage of development, your users, and more.
Here are the experiment types you should consider when running product experiments.
1. A/B testing #
A/B testing is a popular way to test new ideas and validate assumptions. It involves simultaneously deploying two versions of a variable—like a webpage, feature, or workflow—to different user segments and seeing which works best for users.
Using our previous example scenario, let’s say you wanted to test two versions of a new and improved onboarding flow. You can deploy two different versions of the same flow to different users to see which achieves the highest completion rates. One could involve moving the integration CTA to the final step, and the other could remove it altogether. Here’s an example of what it could look like with Tours built with Chameleon 👇
2. Multivariate testing #
Multivariate testing is similar to A/B testing, but it involves testing multiple variables at a time. It’s used when your experimentation involves multiple components that you’re rearranging and combining to find the best solution.
It involves a lot of cross-testing different multiples to find the ideal set-up, but the results are comprehensive and insightful. Here’s a visual created by HubSpot that shows the difference between A/B and multivariate testing.
Let’s say you’re testing the same onboarding flow as the one we’ve mentioned in the example above, but you’ve changed colors, button placement, and step order. Multivariate testing helps you test all the variations of your product experiment to find the best combination.
3. Usability testing #
Usability testing involves testing your product experiments with real users to see if it’s usable and intuitive enough for them to complete their goals.
It consists of giving them a list of tasks to complete in order to observe their process and interactions. Usability testing can help you identify friction points in your product experiment to help validate a new design.
If initial usability testing of your new onboarding flow identifies an issue, you’ll need to make changes to rectify this issue. If further testing shows no more issues, you’re on the right track.
4. Click tracking #
Click tracking helps you identify where your users are engaging with your product and features by monitoring the actions that users take inside your product. Click tracking, or event tracking, looks at actions like clicks, mouse hovering, form field inputs, forms completed, and more.
Monitoring events can help you understand how the user navigates through your product updates and features, as well as what you can do to eliminate friction points and improve the flows.
In our example from above, this could look like prompting users to “find out more” about integration options. Click tracking could reveal that very few users engage with the CTA button, meaning it might be better to hold off mentioning integrations until the user has performed some other actions within your product.
This is where creating custom user segments is really helpful. You could then group segments that share the same characteristics for a cohort analysis of their behavior to be more precise in targeting.
We talked about this in our recent webinar with Userlist on how to use lifecycle segments to retain more users. Watch the webinar recording right here 👇
👀 How to use lifecycle segments to retain users
Not sure how to define event names and properties for the biggest impact? Find out in our webinar where Chameleon and Userlist joined forces to share pro tips on why user segmentation and event tracking are essential to product-led growth.
5. Fake door testing #
Fake door testing is a research and experiment technique that helps you gauge user interest in new features or products. It consists of presenting users with a call to action or lo-fi prototype to see how many people will click on it and try to use it.
Painted door testing, as it’s also known, enables you to validate new features and engage interested customers. It can also be useful for prioritizing product updates—if you’re unsure, ask customers with fake door tests and see what’s most popular.
🦎 For example, Chameleon can help you place in-product fake doors quickly and easily to run your experiments.
However, one thing to consider when it comes to fake door testing is that it’s best to use it sparingly. It can cause disappointment with your users and could diminish your credibility within the product if used too frequently.
6. Beta testing #
Beta testing is a form of feature testing that comes further along in the product experiment process. It involves releasing an almost finished version of your new feature in order to assess its overall effectiveness.
Beta testing assesses your design’s functionality, usability, reliability, and compatibility with real users in order to identify any final issues or changes.
It is also a great opportunity to nurture your Promoters following an NPS survey. This helps build a stronger client relationship as users feel appreciated and included—privileged to be one of the trusted testees.
The benefits of this are two-fold: you get eager beta testers and a pool of engaged customers who you can turn to for further user research, like interviews or usability testing. It’s these users that’ll drive product success by being your number-one advocates.
Run in-product experiments to drive product growth faster #
A good experiment is a SaaS product team’s best friend. It takes the guesswork out of development and helps ensure that you’re putting your time and energy into the right solution.
In-product experiments make feature updates, interface redesigns, and product advancement effortless—you can deploy them quickly and easily across your product to get the insights you need. It saves you time and money and ensures you’re giving users exactly what they want instead of what you think they want.
Remember: Your product development should be based on facts, not theories. Approach experiments with the curiosity that fueled your childhood experiments, and combine it with the knowledge and processes that you’ve developed as a top-notch product manager.