We have been talking a lot about benchmarks with our clients recently. Benchmarking is important, as it provides context to the results that we are analyzing and interpreting; moreover, it is human nature to want to know “how we stack up” relative to the competition. In this three-part series, I’ll provide some commentary on how to evaluate our benchmarking efforts.
We typically see two types of benchmarking that clients want to pursue:
– Industry norms/benchmarks
– Competitive benchmarking within the customer survey (i.e., the customer evaluates both the client and its competitors)
Let’s focus first on industry norms. I see references to benchmarks in the news nearly every single day – these news items generally present the norms as statements of fact – rarely do the studies address the key mechanics of the sampling or qualification criteria. This is not surprising, as the mechanics, while important, do not make for compelling news (as an aside, if you want to read an excellent book on the topic of how the media portrays mathematical principles in non-technical terms, check out the book A Mathematician Reads the Newspaper, by Dr. John Allen Paulos).
In the customer loyalty space, industry benchmarks are abundant – the ACSI and J.D. Power rankings certainly come to mind – and Walker has a number of industry benchmarks that were created as a part of the Walker Loyalty Report series. The key appeal of an externally-sourced benchmark is that it strengthens your position by providing external (i.e., objective) support for your study’s findings and your strategic recommendations.
While many excellent industry benchmarks exist, they may not always fit your particular benchmarking needs. With this in mind, we would advise that you contemplate a few key points when considering the use of an industry benchmark:
1) There is no such thing as a perfect industry benchmark – Research is an inexact science, which is why we state the probability of certain outcomes. Our goal is to manage variance in such a way that we minimize the opportunity for making erroneous conclusions. As such, we need to understand the limitations of the industry benchmark and view the results through the lens of these limitations.
2) Unless you are a part of the benchmark, your data are not perfectly comparable – This does not render industry benchmarks as useless; however, you should exercise caution when conducting direct comparisons between disparate data sets. Looking for general trends is acceptable; however, attempting to conduct precise comparisons is a riskier proposition.
3) Be careful about projecting to “the market” at large – Be certain to “look under the hood” at the benchmark to see how the sample was derived and what it represents. For example, does the sample consist of B2B customers, B2C customers, or both? Are there geographic concentrations we should be aware of? Is the benchmark a result of various industries contributing customer information for inclusion in the study? Having a keen understanding of the sample source and any inherent bias provides the context for how you should interpret – and generalize – the findings.
4) Make certain the benchmark “fits” – As our businesses evolve, there may be a need for comparing ourselves to non-traditional competitors. For example, if a firm’s history is in the manufacture of hardware for use in computer systems but the firm is strategically migrating toward a more service-oriented model, then the firm might consider benchmarking themselves relative to service providers (even outside their industry) as opposed to technology manufacturers.
While external industry norms are valuable, it is also good to remember that we can capitalize on the competitive perceptions of an extremely important set of the overall population – our customers! In my next article, I’ll focus more on the topic of questionnaire-based competitive benchmarking.
Sr. Vice President, Consulting Services & Resource Management