🤔 Challenge: How can Product and Product Marketing teams report more effectively on what is driving users to the product and what friction they’re encountering?
💡 Solution: Pair data from in-app Chameleon Experiences with user behavior data in Heap to eliminate friction, roll out product improvements, and better inform users about product updates.
🏆 Outcome: Collected actionable feedback and detailed bug reports from users during beta tests for new features and UX updates to improve the product ahead of GA release.
SaaS companies store and manage millions of pieces of data. In order to do that, they often rely on what we call ETL (Extract, Transform, and Load) tools, platforms that automate the process of data integration. These complex systems extract data from a source and transform it to meet the requirements of a target system where it will be loaded (like databases, data warehouses, and similar).
Fivetran started from the understanding that conventional ETL tools were not adequate for teams utilizing cloud-based software and storage, and their complex setup often caused project failures. To improve analytics projects, Fivetran created pipelines with zero configuration and maintenance needs to bring data into modern cloud storage.
The company’s product is an automated data platform that allows customers to synchronize and transfer data across multiple clouds. It has six primary features: data movement, data transformation, data security, governance, extensibility and management of data, and finally, data source connectors.
The switch to Chameleon
Fivetran wasn't new to digital adoption platforms (DAPs) before signing up for Chameleon; in fact, they had been using an alternative DAP for both analytics and in-app messaging. Once the team evaluated that it fell short of the level of analytics that they wanted to self-serve out of the product, they made the switch to Heap.
After switching to Heap for analytics, we needed software that could integrate with it.
Chameleon was the answer. Besides the deep integration with Heap, Fivetran was also looking for the ability to set rate limits on the in-app messages they were creating, and for overall ease of use and deployment, which Chameleon ticked all the boxes for.
The team’s expectation was to be able to report more effectively on how users were interacting with the product, announce updates and walk users through them, and finally, collect contextual feedback on user experience.
They were able to do that by flowing data from Chameleon into Heap and vice-versa to accurately target user segments with relevant messaging and then track how they interacted with those by sending events from Chameleon into Heap for funnel analysis.
We can get Chameleon events into Heap so we can report on it, which is helpful from just understanding if an in-app message was effective, resulting in the outcome that we are looking for.
Expanding use cases for in-app messages
By using Chameleon to focus solely on in-app messaging and measuring how and when they were most effective, Fivetran eventually expanded its use case.
What we really liked was the engagement from the Customer Success team. It wasn't a ‘let's upsell you on our different add-ons’, but instead, ‘let's make you very successful with the message you're trying to get across.
Fivetran uses Chameleon-built in-app messages to:
Highlight key features and drive users to perform certain actions
Onboard new users with product tours
Announce and walkthrough UX changes
Improve feature awareness
Collect user feedback to understand intent and behavior
Let’s dig deeper into how they tackled some of those use cases.
Tours to announce feature betas and recruit users to opt-in
One of Fivetran's main use cases is telling users about new private preview connectors, the early access version of Fivetran data connectors that are made available to select customers. These are designed to allow customers to test and provide feedback on the feature before it’s released.
In other words, their goal is to recruit relevant users for beta tests, for example, those that use a database that can be integrated as a private preview connector. Besides accurate targeting capabilities, rate-limiting was also necessary to enable this effectively.
If somebody logs in after we've made a big UX change, we will let them know by showing an on-screen message saying 'Hey, we've changed some things, do you want to take a tour?
From this interaction, users can trigger a Chameleon Tour that guides them through the changes.
Welcome pop-up built with Chameleon for users to self-serve a tour of product updates.
The need to keep users up-to-date with product changes extended itself to every new release, so Fivetran also started to use Chameleon to send targeted notifications about minor product updates when relevant for the users.
Microsurveys to understand user intent and behavior
Looking at raw data on how users are interacting with your product cannot shed light on questions about their intent, like what actions they were hoping to take, or what they're trying to accomplish.
Fivetran uses Microsurveys to ask simple questions that are often difficult to find answers to through raw user data.
We are able to quickly ask users some easy questions to get insights into areas we may not have a good understanding of. It can be as simple as 'Hey, why are you logging in today?' – not necessarily to get them to do something, but to understand what a user is trying to accomplish when they come to Fivetran.
Dropdown Microsurvey built with Chameleon to understand user intent.
The integration with Heap is essential in this analysis. Through Heap, Fivetran can see what users are doing before they report an issue, or what actions they are taking in order to accomplish what they note as their intent.
The other thing we could do, which was really cool, was take the survey results and see how they impacted user behavior. For example, if a user says they came in for a certain issue, we can then see what they did after that and how that compares to other users.
Reducing product friction with a simple question
From looking at user behavior in Heap, the product team at Fivetran noticed that many users were deleting data pipelines (connectors), and couldn't understand why. Using Chameleon Microsurveys, they targeted these users with a simple question: “Why did you delete that?”
From the responses, they found a gap in their product: users had to delete schemas to rename them, and that was skewing their data and causing friction.
This was something that we were not aware of, but it was sort of the "aha!" moment of 'oh, we should maybe look more into renaming since customers are deleting a lot there and it skews our data.
As a solution, Fivetran began pursuing the enhancement.
📈 Result: Impact on product roadmap
Besides being able to interact with users to identify points of friction and find solutions, Fivetran found that collecting contextual user feedback allowed them to:
✅ Measure customer satisfaction to find other areas for improvement
✅ Report on the impact of product changes more actively and accurately
✅ Add more accountability for product marketing to make decisions
With Chameleon and Heap integration, we could really start to hold Product Marketing accountable to measuring the impact and reporting on it by using out-of-the-box reports within Heap that we built using all the Chameleon event tracking data.
Besides these product-specific inputs, Fivetran is also using Microsurveys to measure customer satisfaction in a few other areas, like in regards to their pricing and ease of use of the platform.
Using two of the three surveys that our UX research team has identified, we're using Chameleon to ask users who are on our usage page how well they understand and feel supported in understanding how they're billed. The other one is whenever a customer sets up a connector for the first time for specific stores, asking them how easy it was, so we get a consistent ongoing view of areas to improve.
Feedback collection during beta tests for successful product launches
Fivetran is currently undergoing a significant UI redesign and has implemented a beta test, where users who have opted in can toggle on and off between the current and new UI. With a feedback link on their dashboard that triggers a Chameleon Microsurvey, Fivetran can ask the users for input and feedback.
We found 10-15 bugs that, maybe, if we didn't use Microsurveys, we wouldn't have noticed before the GA release. So I very much anticipate it to be a very smooth release since we have over 1000 customers on the new optional opt-in beta, and they've had the ability to report issues.
📈 Result: Identified 15 product bugs ahead of GA release
Fivetran anticipates a smooth transition to the official release of the UI redesign because Chameleon also allowed them to:
✅ Receive 46 pieces of feedback, which have aided in addressing and fixing issues
✅ Understand what users are doing before and after reporting an issue through the integration with Heap
✅ Opt-in 1000 new customers into the beta with the ability to report on issues
To sum it up
The partnership between Fivetran and Chameleon started from their need to find an in-app messaging platform that could integrate deeply with Heap, but it quickly evolved as we helped Fivetran expand their use case for our products and analyze data from Chameleon Experiences.
Besides the results that Fivetran was able to report on from using our Tours, Tooltips, and Microsurveys, they have now expanded into using Launchers (in-app checklists) and are eager to measure their impact (and so are we).
Launchers were a pleasant surprise! We didn't have these in our old tool and we never used anything like it to help users understand the value of different functionalities during their onboarding experience. We'll continue to experiment with Launchers and hope to see them help with nudging users to take specific actions.