Menu
Walker Information
Helping you put the customer at the heart of every decision.

An overall review and critique of net scores, Part 1

I had the opportunity recently to discuss the issue of net metrics with a client; given the popularity of the topic of net scores (thanks in large part to the Net Promoter metric), I thought it would be beneficial to share our thoughts on the topic in this blog. In this first installment, I’ll talk about some of the methodological issues you should consider when thinking about net scores.

Net scores are composite scores that are defined by taking the difference between two scores; this can be from two different questions (for example, the Top 2 score of Quality minus the Top 2 score of Value) or from within a single question (for example, the Top 2 score minus the Bottom 2 score).

Net scores are touted to be effective due to their ease of understanding; assuming this is true, we give up a couple of key statistical elements that we believe most clients would find troubling from a target setting and/or actionability perspective:

  • The margin of error around net scores balloons as a result of the need to account for variance in both portions of the metric. The result can be a metric with a margin of error so wide as to be meaningless. For example, if our net score is 70 with a margin of error of +/- 20, this means our actual results fall within a range of 50 to 90. The only way to counteract this is to dramatically increase the sample size. Increases in sample can, of course, have cost implications, particularly depending on the method of data collection employed.  This limitation is particularly critical, given the desire to set performance targets – a wide margin of error can prove to be not only difficult, but also de-motivating.  
  • Net scores are just that – net. As a result, when we see movement, we cannot discern what portion of the score is moving. Consider the following scenario
-Two groups, Entity A and Entity B, each move their net score from 40 in period one to 60 in period two. From a net score perspective, they are doing equally great.
 
-If we decompose the net score into its component parts, we see a much different picture:

 

Wave 1

Wave 2

Group

Top 2

Bottom 2

Net

Top 2

Bottom 2

Net

Entity A

65%

25%

40%

63%

3%

60%

Entity B

50%

10%

40%

77%

17%

60%

Given these scores, do we really think both groups are making the same progress? In this example, Entity A is making progress in reducing the number of Bottom 2 scores, while Entity B is growing the Top 2 Box scores. The point is, we need to understand how each element is moving relative to the other to truly assess who is improving and who is declining.

  • The nature of a net score is, by design, an aggregate level indicator. It has no utility in helping us to profile individual client relationships (unless we combine, as NPS does, individual score points into categories like Promoter and Detractor). 
  • Finally – and perhaps most problematic from an actionability perspective – the utility of net scores (when using a single metric) is limited to the aggregate level. Unless we arrive at categorical definitions (again, like the NPS approach), the data by itself can have constrained value.

The whole premise of using net scores is that they are easier to understand – there is a school of thought that says that any metric that requires some level of calculation is, by definition, more difficult to understand. The scenario cited above reinforces this relative to net scores; the same could be said of indices as well. As with most metrics, you’ll need to consider your culture and the organization’s ability to assimilate and utilize any metric – after all, measurement for the sake of measurement is neither strategic nor a wise use of the firm’s time and resources. Creating a bias for action is the key.

In my next entry, I’ll review some common ways our clients use metrics and how these drive action. In the meantime, I’d be interested in your thoughts – what metrics do you use in your organization? Do they resonate with your co-workers? Do they prompt action?

 
Mark Ratekin
Sr. Vice President, Resource Management and Consulting Services


Share your thoughts

Your email address will not be published. Required fields are marked *