Walker Information
Helping you put the customer at the heart of every decision.

Benchmarking Customer Benchmarks, Part 2: Survey-based Benchmarking

In my last post, I talked about the use of industry benchmark/norms from external sources as a tool in our customer analytic toolbox. However, it’s important to recognize that our customers can be key sources for competitive intelligence. This is particularly relevant for industries in which multi-sourcing is the norm.

We routinely help clients with survey-based benchmarking. The typical approach we employ involves three steps:

          Identifying the benchmark of interest – Ordinarily, we’ll gather the list of all providers the customer is familiar with and will then focus on either the one the customer is most familiar with or a random selection among the possible selections.

          Getting the relevant ratings – This can be accomplished in a couple of ways – we can either rate each firm in an absolute fashion (and then look at the gap scores), or we can assess the relative difference. Here are a couple of examples:

o   Absolute method – This method looks at each company in a stand-alone fashion:


Q1. On a scale of Excellent, Very good, Good, Fair or Poor, how would you rate ABC company’s product in terms of overall quality?


Q2. On the same scale of Excellent, Very good, Good, Fair or Poor, how would you rate XYZ company’s product in terms of overall quality?


To get the benchmark, we look at the gap in scores between Q1 and Q2.


o   Relative method – This method does a direct comparison between two companies:

Q1. Thinking in terms of overall product quality, how does ABC company compare to XYZ company? Would you say Much better, Somewhat better, About the same, Somewhat worse, or Much worse?

          Analyzing (and reporting) the findings – The analytic steps we take can be as simple as reporting the frequency of company mentions and as complex as running advanced models using the gaps scores as the inputs into the model.

This seems pretty straightforward – and it is. However, like most good survey designs, the most critical work begins before the first interview is conducted. During the design process, we have to ask a critical question – who should we compare ourselves to?

In a customer loyalty context, the conventional wisdom says you want to compare yourself to other firms that provide similar products and/or services. Consider, however, the following scenarios:

          What if your firm has few competitors (think, for example, about some public utilities)?

          What if your firm’s customer base typically sole-sources (due to a fixed investment that is required in establishing and maintaining relationships, for example)?

          What if we are conducting a study focused on the overall health of the customer relationship, but the products and services offered by your firm have distinct and separate competitors?

          What if your firm is part of an industry with lackluster performance (airlines, for example)?

I could argue that any of these situations suggests that same-industry competitive benchmarks would likely produce either incomplete information or information of questionable value.

Here are some questions to ask yourself when considering survey-based benchmarking that might light the path toward improved competitive assessments:

1)      How do our customers buy our kinds of products and services?

2)      What companies would our customers identify as providing an ideal customer experience? Are there things we can learn from these experiences?

3)      What is our market share? Do we have any close competitors?

4)      Do our customers multi-source? If not, do they have perceptions of our competitors that can credibly be gathered and analyzed?

5)      What is our strategic direction? Are we moving in a new direction? What other experiences might our customers have that have relevance?

If you don’t have clear answers to these questions (or if there is lacking consensus among your engagement team), it might suggest that your research objective either lacks clarity or your intended audiences for the information may have different expectations on how (and to whom) to benchmark. To provide clarity around the intent and expectations is critical before we start the study in earnest; with this in mind, it might make sense to conduct some qualitative interviews prior to designing and launching your survey.

Like the industry benchmarks, there is no perfect (or one-size-fits-all) approach. To optimize the information you can glean from the data, you need to do your homework carefully, including having a clearly defined business objective that is driving the research questions.

In my final segment, I’ll share some other questions that will aid you in evaluating not only benchmark data, but your survey in general.

Mark Ratekin
Sr. Vice President, Consulting Services and Resource Management

Share your thoughts

Your email address will not be published. Required fields are marked *