How CX scores can be too good to be true | Clarasys

To continuously improve and remain market relevant you must understand your customers’ needs and expectations. This starts with understanding the quality of your products and services, and the experience created for your customers.

This isn’t a new concept. Two well-established methods of gauging customer experience and understanding the quality of your offering are Net Promoter Score (NPS) and Customer Satisfaction (CSAT)*. However, these metrics can be open to misinterpretation and lead to incorrect inferences about your customer offering.

For your organisation to have representative customer experience metrics, the inputs for your scoresmust span the breadth of your customers’ experiences and must be statistically significant**.

“It ain’t what you don’t know that gets you into trouble.
It’s what you know for sure that just ain’t so.” – Mark Twain

This is the crux; your CSAT scores could be causing you to make suboptimal decisions.

Let’s do a sense check

If your organisation uses these scores, then does your organisation…:

• Have exceptional NPS/CSAT scores?
• Have great reviews but a declining customer base?
• Have a low percentage of users providing feedback responses?
• Evaluate performance using NPS/CSAT scores?

If you answered yes to any of the questions above, then your organisation may have a source of inputs which cause an unrepresentative view of reality: Incorrectly used inputs create blind-sports that hamper one from staying market-relevant.

We’ve seen how this can manifest in a subscription-based organisation, offering services to satisfy thousands of customers with complex and variable needs.

High NPS and CSAT scores create a belief that the organisation is doing well, when in reality it is hamstrung by the devastating knock-on effects of its unrepresentative NPS/CSAT scores: customers are not understood, complacency sets in, targets are missed, blame is shifted… customers leave.

This has been underpinned by good, honest intentions to do the best thing by its customers, but the execution results in a distortion of reality. See the figure below for a visual illustration.

Could this be affecting your organisation?

We’ve observed that organisations which have ‘good’ but unrepresentative scores are likely to have a sample which is induced by:

• Employee performance measured by these metrics
• Feedback gathering processes which can be gamed
• Have scores which could be used to flatter management rather than identify improvement opportunities

The above can lead to perverse incentives. For example, if an employee’s performance is related to the CSAT score they receive from their customers, they’re very unlikely to ask the customers who they believe is unhappy to give feedback, skewing the range to be flatteringly positive whilst reducing the number of potential responders. That red curve from the visualisation forms.

So how do you use these metrics correctly?

My maths teacher bluntly described a ‘function’ to me: rubbish in → rubbish out.
NPS and CSAT scores are functions and require inputs, real customers who represent a range of experiences. If your customer input sample does not reflect reality, then you guessed it… rubbish in → rubbish out.

Some CSAT clarity

We recommend applying for following mindsets to create a representative view of reality If you believe the correct metrics to measure your organisation’s success requires inputs similar to CSAT and NPS:

• Being deliberate about the feedback sample:

Such as automating the feedback; set targets for number of customers asked not just a score target; analysing a random sample of your feedback to repeat results.

• Customer-needs focussed:
  Identifying the salient groups of your customers and tailoring metrics to unearth their needs and what’s most impactful to their experience, not just a generalised score.
• Regularly challenging the scores and probing for the next improvement opportunity: asking specific questions to use your metrics to answer or set up unbiased, representative ones to do so.

By bearing these considerations in mind, your customer experience metrics will allow you to avoid believing scores too good to be true, and help your organisation identify what matters most to your customers.

So, on a scale of 0 – 10 how likely are you to recommend your organisation’s approach to understanding customer experience?

* Other metrics exist, but this blog focuses on NPS and CSAT.

** A sample of responses large enough to ensure the result was not due to a sampling error.