Retail Decoded with JD Trask, CEO and Co-Founder, Raygun
In our last column, we explored Google’s customer experience score as one of the best ways to measure the quality of your digital storefront. Today, I want to talk about the other side of the equation: the metrics to avoid or treat with caution.
When it comes to measuring customer experience and satisfaction, businesses have traditionally relied on self-reports and surveys, methods that are prone to bias and misinterpretation. As larger sections of the customer journey move into digital spaces, new tools have sprung up to capture information about real customer actions and interactions, quantifying how fast, successful, and enjoyable the experiences we deliver really are. By combining this objective, quantitative data with traditional data about the customer’s thoughts and feelings, we can see the full picture for the first time.
If you’re not sure how to achieve this level of insight, you’re not alone: only 38 percent of executive leaders say their organization does well at measuring customer experience, and 27 percent say they struggle (with the rest falling somewhere in the middle). Measuring the wrong indicators or measuring too many variables can easily become misleading and overwhelming, or even contradictory. Sure, data is powerful, but only if it’s relevant, accurate, and actionable.
Now, let me preface by saying that if you are using either of the metrics, we talk about today, don’t panic — they’re not innately bad or harmful and are actually still helpful sometimes. Plus, measuring something is better than measuring nothing. It’s just that these metrics, while popular, are often overused or misapplied. Let’s see why.
Averages
An average is the best way to understand where the majority stands, right? Well, sometimes. Averaging works well when your data set is clustered within a relatively small range. But when it comes to measuring digital experience, an average can actually work to conceal the reality of large portions of your customer base. Because experience signals like page load time tend to fall across a broad range, it’s easy for a few outliers to skew your average and lead you astray or for the outer values (which need the most attention) to be hidden.
At Raygun, we use the median as a better indication of where the ‘typical’ customer actually sits because your median, by definition, will show where most of your customers are.
We also consider the P99 or the quality of experience for the 99th percentile of users. (The 100th percentile is better left out of the equation, as this is where total timeouts and bots tend to throw the numbers out wildly). Once you’ve set the upper limits of what your team terms an acceptable customer experience, your P99 shouldn’t exceed this. Compared to your ideal time, this is going to be slow, but you still want to keep this closer to five seconds than 25. Tracking your P99 ensures that even the laggiest, buggiest, most frustrating experiences you deliver are still within reasonable parameters. Keeping your P99 in range acknowledges that we can't guarantee perfection every time, but we can still keep ourselves honest.
Net Promoter Score
NPS remains deeply ingrained in many businesses, processes, and industries. It’s the most popular customer experience metric and continues to have valuable uses. But NPS has become increasingly controversial, and Gartner has predicted that 75 percent of organisations will no longer use it to measure customer service and support by 2025.
I can’t say whether that will come true — there are good reasons why NPS became an industry standard, and a strong focus on customer feedback has raised standards and reminded businesses why they exist (hint: it’s the customer). However, many of us over-rely on the famous score, and in doing so, overlook the risks and misrepresentations NPS can introduce. NPS is flawed in a number of ways; it amplifies the most extreme user experiences, so isn’t representative of the majority. It can be challenging to accurately attribute or improve NPS scores, because better scores could be a matter of working on your product, service, price, or even just changing the market you’re selling to. You could deliver a slow and buggy website experience followed by an excellent support interaction and still score a nine based on the last contact the customer had. If you’re only relying on NPS in that scenario, you’ll never flag or address the bug, and other customers who encounter the same issue might just quietly leave.
NPS is a number, but it also gathers qualitative data, and this can be one of its most productive uses, whether it’s relaying service feedback and giving upsell opportunities to your customer-facing teams, sharing feature requests with your product team, or passing promoters to your marketing team as potential testimonials. In short, NPS remains relevant as a large-scale, subjective indication of how your customers are feeling, giving them the chance to tell you what’s on their minds. But it doesn’t stand alone as an objective measure of experience, particularly for your technical teams.
So, what else should I use?
Nobody likes to hear problems without any suggested solutions. Raygun customers often ask us about useful and clear ways of measuring experience, and it’s a question we’ve given a lot of time and attention to. To help businesses find answers, we’re currently putting together a guide to digital experience monitoring, sharing ten metrics that can help you see through your customer’s eyes. If you’d like a copy, just fire me an email at jdtrask@raygun.com, and I’ll make sure you’re on the list to receive one. Otherwise, I’ll share more on this topic with you in this column soon.
