At Chattermill, our mission is to deliver precise and actionable insights, enabling our partners to have a complete, objective understanding of their customers every day. We analyze customer feedback across customer touchpoints for organizations to harness customer insights at scale, boost customer loyalty and brand advocacy.
Today we’re taking another step forward in that mission. We are proudly announcing our latest product: the “insightfulness” filter.
Part of Chattermill’s paradigm is in using the latest research in artificial intelligence to empower our customers. To that end, we took a deep dive into the evolution of preference statistics to create a machine learning solution to evaluate the “insightfulness” of each piece of feedback. In turn, our model shortens the time gap between manually reviewing comments and arriving at actionable insights, by surfacing the most constructive pieces of criticism. This blog will go over the story of its construction, from data to deployment.
Collecting preference data
The “insightfulness” of a comment can be equated to a preference. I use preference in the economic sense here, meaning “the judgment used to select one item over its alternatives”. In our case, we’re describing the preference of reading useful criticism instead of one-word answers or incoherent sentences. How do we appraise such a preference in a way that we can automate?
We begin by assuming that preference is quantifiable. That’s a large and consequential assumption - what allows us to take such a statement to be true? Well, if we can compare our preference between two or more items, then each item could be said to have some inherent “preferability” score, where higher scores will get picked more often. Then the question remains “how do we measure that score exactly?”. Fortunately, we can use some tried-and-tested methods.
The most straightforward way is a pointwise system (for instance, “what would you rate this item out of 5 stars?”). By averaging a dataset of ratings, we will attain imperative values of preference. However without any absolute metrics for reference (what is the difference between 0 and 1 stars exactly?) this data tends to be noisy, and therefore unreliable without very large quantities of labels.
Alternatively, we can measure preferences relatively through pairwise trials. In these trials, you repeatedly present users with two items and ask which they prefer. Over time you can begin to resolve which items are more likely to beat others.
Statistical models can evaluate these trials to produce real-value scores for each item (for instance the Bradley-Terry model). Models like this enable Netflix and Youtube to understand the inherent “user interest” of videos in a feed, and consistently surface relevant content to their users.
The story of Insightfulness
Our customers described the preference problem at hand: “a lot of the feedback we receive is not constructive. It takes a great deal of time to skim through less insightful comments such that we can arrive at actionable insights”.
Our data science team’s solution: based on the assumption that each piece of feedback has some inherent “insightfulness” score, if we filter out feedback with low numerical values, we’re left presenting only insightful content to our customers.
We measured insightfulness in the pairwise trials which were described before. Using pairwise labeled data, we trained a state-of-the-art machine learning model to predict these scores and uncover the inherent insight of each comment. The results were impressive. We can take a subsample of score ranges to demonstrate; the following examples had the lowest model values:
This image speaks for itself. What is the use in reading feedback like this? The worst offenders are incoherent. The others are one-word answers which can already be summarized by our theme/sentiment labels.Meanwhile, the distribution of top scorers is quite different. The writers are not only more descriptive with their comments, but also provide constructive criticism on what to improve:
Using the insightfulness filter, we have optimized our customer’s time when reviewing comments by surfacing only the most meaningful feedback received. When combined with dashboards and feedback widgets, the functionality of the new filter becomes a powerful tool in their arsenal. We’re very excited to offer our users a new and improved way to view their customer data!
Concluding
Customer preference can be considered a real, measurable quantity. This post explained some of the data science behind collecting and evaluating it, as well as demonstrating the design behind our newest product: the insightfulness filter. At the end of the day, we know that not every customer comment is a winner - now you can separate the best from the rest.
We’d love to hear your thoughts on this article! Feel free to reach out at thomasmetcalfe@chattermill.io
Would you like to filter insightfulness from your data? Get in touch with us and book a trial today.
