| Customer Feedback Analysis Walker's analytic experts will present ideas about gathering customer feedback to deriving insights from it that will positively impact your customers and your business. |
In previous blogs, we have reviewed findings-to-date, regarding what specific elements of the survey process have been shown to be influential on response rate. If we take a step back from the specifics, at the heart of the response rate issue, is respondent motivation. How do we get respondents to want to take a survey? In a recent article by Pete Cape, “Understanding Respondent Motivation,” he provides a historical review of how psychological theory on human motivation can help us tap-into motivating survey respondents. This week, I will start with his review of the Behaviorists, relating how Behaviorist theory can provide insight to the response rate issue.
Behaviorist Theory in a Nutshell
· The behaviorists provided the concepts of reinforcement, punishment and extinction.
o Reinforcement concerns strengthening a desired behavior as the result of either experiencing a positive condition (Positive Reinforcement) or eliminating a negative condition (Negative Reinforcement).
o Punishment concerns weakening behavior as a result of experiencing an aversive condition (Positive Punishment), or removing a positive condition (Negative Punishment).
o Extinction occurs when the reinforced behavior is no longer effective.
· A primary criticism of behaviorist theory is that it provides a mechanistic view of human behavior, which is limited, as the motivation behind human behavior is complex.
How does Behaviorist Theory Aid Understanding of Response Rate?
Sometimes you can learn the most from the criticisms of a theory. Respondents are unlikely to be sustainably motivated to complete a survey by making any single improvement, or removing any single negative component, at one point-in-time. For example, providing a deadline by which to complete the survey has been shown to significantly impact response rate; however, a deadline alone is not likely to sustain response rate, rather there are many other influential factors. Instead of focusing on any one factor, consider the entire survey process-
· Remember that the process starts before the respondent chooses/declines to take the survey- creating a survey that is easy for the respondent to take, pre-communication, personalizing communication, scheduling reminders, etc.
· And, the process continues after the respondent completes the survey- communicating and acting upon findings, showing and proving that the feedback matters.
Next week, we will start to examine psychological theories that assume a more humanized-approach, exploring how understanding human needs can help us tackle declining response rates.
Amy Heleine
Director, Marketing Sciences
Reference
Cape, P., “Understanding Respondent Motivation,” Survey Sampling International White Paper, 1-17.
Science has advanced mainly through failed hypotheses. New theories, ideas, or inventions are based on the invalidation of a previous hypothesis. This type of advancement is also true in other arenas, including business. However, most people focus on confirming hypotheses and spend little, if any, time looking for contrary evidence. I think there are a number of reasons for this:
- We have an emotional investment in the hypothesis and want to be right. This is especially true when we created the hypothesis or when our job is seemingly dependent on the hypothesis being true (which is hardly ever true, even when it seems to be).
- Our boss or executive leader is a big proponent of the hypothesis, and we don't want to contradict them.
- We created a bad hypothesis that really can't be invalidated.
- We don't even consider that our assumption or belief should be tested and skip creating a true hypothesis. We treat our assumption as a statement of fact and then go about putting together all the evidence that supports it without even considering contradictory evidence.
In a business context, focusing too much attention on confirming our hypotheses can lead to lethargy, a culture of acquiesence, and poor business decisions. Most hypotheses are created from our assumptions about what is true or what has been true in the past. So focusing only on affirming hypotheses will result in an organizational culture that doesn't question assumptions and continues to do what it's always done.
This can be espescially dangerous when developing customer insights or customer strategies. Faulty assumptions and flawed, stagnant processes are most prevalent where the customer is involved. Hypotheses are generally formed by our perspective and not the customers'. We fully expect the customer will love our new product and find all sorts of evidence to support that view. However, customers hate it, and the product fails. Or we assume our service is far superior to our competitors and customers will not be swayed by their advancements. Then we find that customers are defecting to the competition in droves.
I would encourage everyone, especially those responsible for developing customer strategies, to do five things:
- Create a culture of testing and inquiry. It is essential to moving your company forward.
- Develop good hypotheses based on your assumptions.
- Know what it would require to disprove the hypothesis.
- Try to find good, solid evidence against your hypothesis. The supporting evidence is much easier to find.
- Be willing to challenge the common wisdom of your company or leadership when the hypothesis is shown to be wrong. This can be uncomfortable, but if you do it respectfully and with the support of strong evidence, the message should be accepted. It also helps to have a few other colleagues in your corner!
Disproving a hypothesis can be uncomfortable, especially when it's based on a widely held belief. But if it's not true, and you don't do disprove it, someone else will - either a colleague or a competitor. And you may miss the chance to make the next big advancement for your company, your industry, or maybe even the world.
Troy Powell, Ph.D.
VP, Statistical Solutions
"The goal of customer satisfaction is thoroughly corrupted when it transforms from the altruistic service of better pleasing customers to the unfettered purpose of serving oneself."
- Dennis Murphy and Chris Goodwin
"Satisfying no longer" Quirk's Marketing Research Review, August 2009, p. 55
The October issue of Quirk's published the last of a three-part series on how satisfaction measurement has gone awry and ways to fix it. I thoroughly enjoyed this series and would encourage anyone who administers or uses data from a survey-based customer feedback program to read it.
Of all the problems with traditional customer listening programs, there is only one thing the authors claim will thoroughly corrupt a program - tying feedback scores to corporate incentive plans or compensation. This may seem counter-intuitive. Including customer scores in an incentive program helps focus the organization on customers, right? Ideally, yes. Practically, not so much. It focuses the organization on the score, but not usually on the customer.
The authors provide three key reasons why tying incentives to customer feedback scores is dangerous:
- The goal of the survey shifts from pleasing the customer to pleasing the organization. If you currently use survey scores in your incentive plan, consider how much time you spend setting, explaining and defending the goals. And this time increases exponentially if you miss a target, which is exactly when you should be talking to customers to see why scores are down instead of defending the numbers internally.
- Those responsible for the survey become the "police" instead of the "partners." The time you spend with internal stakeholders is more often focused on defending the metrics and methodology than on helping the organization internalize and act upon the customer feedback.
- You actually give executives an incentive to question the research and results. Making it part of the compensation plan brings high-level attention to the customer survey, but actually encourages executives to question and nit-pick the details instead of focusing on the big picture.
What should we do about this? As customer advocates, we definitely need ways to motivate employee action that benefits customers and the organization, but we need to think more creatively on how to do it. Incentive plans are an easy answer, but not the best answer. And given the amount of money companies spend on these plans, a little extra thought is worth it.
One key way to motivate an organization around customer feedback is to illustrate the impact of taking action on customer feedback. We often think of this as a statistical linkage such as "improving loyalty by 5%, will increase revenue by 2%." That type of business impact analysis is definitely important and useful. But it's equally useful to have a customer service department that listens to their customers' complaints, fixes a process, and realizes a 10% decrease in cost. Until we can illustrate the real business impact of using, acting upon, and improving customer perceptions, we will always have a problem motivating people to take it seriously.
I don't have many other solutions to share here, but I will post follow-ups with any ideas that are generated by this topic. Until then, think carefully about what you can do with your customer feedback program to motivate your organization to listen. Part 3 of Murphy and Goodwin's series provides some helpful answers.
Troy Powell, Ph.D.
VP, Statistical Solutions
Over my last two blogs, I have taken a decidedly apolitical view of some of the data used in the debate around healthcare reform in the U.S. and provided some commentary on why the data used may be insufficient to thoroughly explain the current situation in a comparative context. Interestingly, these same five shortcomings can also have a detrimental effect on our ability to motivate customer-focused change in our organizations. In this final entry, I will recap my criticisms of the data on healthcare reform and talk about what we should take away in our daily roles of analyzing and interpreting data in order to help our organizations become more customer-centric.
The five criticisms, and what we can learn, are as follows:
1) What is the underlying question we are trying to answer?
We often hear from clients that their primary goal is to measure loyalty; I respectfully disagree. Simple measurement is not enough – we need to consider the key business issue our company is dealing with and how customer strategies impact that issue. This, of course differs by customer (or, at a minimum, by industry).
In a generic context, our goal should be to threefold - first, we need to understand how customer loyalty can help us to maximize the financial health of our company. Second, we need to understand what customer experience gaps exist in our environment that prevent us from realizing these financial gains, and third, we need to establish processes and procedures that enable us to systematically close these gaps. In undertaking these three steps, we will enjoy some collateral benefits:
2) Is the outcome metric the right metric?
There is a lot of debate about the metrics – loyalty, NPS, satisfaction, value, quality; the list goes on and on. Here’s the harsh reality – there is no one right answer! All of the metrics just mentioned can have some utility in a customer loyalty initiative; the key is to align the right metric to the business question so that the results are reliable, actionable, and resonate in our firms.
3) The role of exogenous variables in our analysis
In the healthcare reform example, I talked about how exchange rates can impact how we view the per-capita spend on healthcare. These types of factors that are outside our environment (and, by extension, outside our control), can have a meaningful and material influence on how we interpret our results and make change over time. While we cannot control these factors, we can often control for them in our modeling. It is wise, therefore, to be looking ahead to determine what factors we might be facing in the near future. Examples include:
4) Selection bias
The cornerstone to any analysis is the sample that is used to build the data set. It is imperative that we have well thought-out sampling plans. For example, some items to consider include:
5) Overly simplistic models
Albert Einstein said “Everything should be made as simple as possible, but not simpler.” The same is true in terms of our customer loyalty efforts – we need to align the rigor of the data with the gravity of the business question at hand. Sometimes this will be as simple as asking a handful of questions and reporting top-two box scores; other times, it will require a highly complex sample and survey design with hyper-complex statistical analyses.
What won’t generally work is an overly simple, one question survey with an open end attached. You will get data (sometimes a lot of it), but you will spend more time trying to find the story in the data than if you had a more rigorous tool.
The idea of starting with the end in mind often works – if we can identify what information we want to convey, it helps us to properly design the program in total.
So, to tie it all up – the prescription for a sound analysis of customer loyalty is alignment of loyalty to corporate objectives, careful planning to make sure that the metrics sync with the business objectives and use of proper techniques.
I hope you found some value in this analysis – if you have questions or comments, please let me know.
Mark Ratekin
Sr. Vice President, Consulting Services & Resource Management
The five criticisms, and what we can learn, are as follows:
1) What is the underlying question we are trying to answer?
We often hear from clients that their primary goal is to measure loyalty; I respectfully disagree. Simple measurement is not enough – we need to consider the key business issue our company is dealing with and how customer strategies impact that issue. This, of course differs by customer (or, at a minimum, by industry).
In a generic context, our goal should be to threefold - first, we need to understand how customer loyalty can help us to maximize the financial health of our company. Second, we need to understand what customer experience gaps exist in our environment that prevent us from realizing these financial gains, and third, we need to establish processes and procedures that enable us to systematically close these gaps. In undertaking these three steps, we will enjoy some collateral benefits:
- We will understand where we have financial security (and risk) in our customer base.
- We will understand who our most valuable customers are.
- We will understand what potential revenue gains exist among our current customers.
2) Is the outcome metric the right metric?
There is a lot of debate about the metrics – loyalty, NPS, satisfaction, value, quality; the list goes on and on. Here’s the harsh reality – there is no one right answer! All of the metrics just mentioned can have some utility in a customer loyalty initiative; the key is to align the right metric to the business question so that the results are reliable, actionable, and resonate in our firms.
3) The role of exogenous variables in our analysis
In the healthcare reform example, I talked about how exchange rates can impact how we view the per-capita spend on healthcare. These types of factors that are outside our environment (and, by extension, outside our control), can have a meaningful and material influence on how we interpret our results and make change over time. While we cannot control these factors, we can often control for them in our modeling. It is wise, therefore, to be looking ahead to determine what factors we might be facing in the near future. Examples include:
- Changing economic conditions;
- Regulatory changes in your market;
- Competitive forces – new entrants, exiting competitors, supplier influences, resource constraints, etc.
- Changing consumer tastes; We recommend taking a fresh look at your program every couple of years to see how things are changing in your environment (and we need to revamp in the program to keep it fresh, relevant, and impactful).
4) Selection bias
The cornerstone to any analysis is the sample that is used to build the data set. It is imperative that we have well thought-out sampling plans. For example, some items to consider include:
- How do we define an account – one person or many people? One customer organization or entities within that organization?
- Should our sample mirror our revenue composition? If so, do we weight results, control the sample, or both?
- Do we have access to (and can identify) all relevant customer contacts? Should we differentiate by decision role, for example?
5) Overly simplistic models
Albert Einstein said “Everything should be made as simple as possible, but not simpler.” The same is true in terms of our customer loyalty efforts – we need to align the rigor of the data with the gravity of the business question at hand. Sometimes this will be as simple as asking a handful of questions and reporting top-two box scores; other times, it will require a highly complex sample and survey design with hyper-complex statistical analyses.
What won’t generally work is an overly simple, one question survey with an open end attached. You will get data (sometimes a lot of it), but you will spend more time trying to find the story in the data than if you had a more rigorous tool.
The idea of starting with the end in mind often works – if we can identify what information we want to convey, it helps us to properly design the program in total.
So, to tie it all up – the prescription for a sound analysis of customer loyalty is alignment of loyalty to corporate objectives, careful planning to make sure that the metrics sync with the business objectives and use of proper techniques.
I hope you found some value in this analysis – if you have questions or comments, please let me know.
Mark Ratekin
Sr. Vice President, Consulting Services & Resource Management
In my last blog, I reviewed some data from the World Health Organization on the amount spent on healthcare in the U.S., Canada, Sweden and Japan. The results of the data were counterintuitive – the higher the spending on healthcare, the lower the life expectancy.
As good analysts, we should scrutinize any data to ensure that it is not only appropriately used (and that any limitations to its use are noted and considered when making recommendations), but also that it adequately addresses the core business question. Of course, for this to occur, we must have a well-defined, properly framed business question.
I set out a challenge for our readers to consider what limitations existed in the data and how this might constrain what conclusions we could make about the linkage between healthcare spending and life expectancy. In this blog, I’ll review five factors that could impact how we interpret the data.
1. What is the underlying question we are trying to answer?
The first issue that creates confusion (and, potentially, misinterpretation of the data) relates to the underlying question at hand – is the goal in the healthcare debate to increase life expectancy overall, to minimize the expense of healthcare, to get coverage to more people, or something else? Of course, like the business questions we face daily, there is not a single answer; I would suggest, however, that pundits are trying to answer too many questions with a single data set.
2. Is the outcome metric the right metric?
The second issue relates to the core outcome metric being employed – mean life expectancy. While longer life expectancy is good, it does not tell us the whole story; to know that, we must understand the underlying distribution of the data. Consider the following data distributions:
Graph 1 Graph 2
Graph 3 Graph 4
The shapes of these distributions clearly suggest differences at play in the data; they do, however, have one thing in common – their mean value (78 in this case).
Let’s explore what these might mean from the perspective of life expectancy – how would we interpret each of these?
Graph 1: The variation around the mean is fairly constrained; this suggests that most people will live within a fairly tight range around 78 years.
Graph 2: The data skew toward a lower life expectancy (roughly 58 years). What this means is while the average life expectancy is 78 years, many people (about one-third) will not live past 63 years; on the other hand, there are a great volume of people that live much longer lives, which effectively increases the average.
Graph 3: This is a normally-distributed set of data with wide variation around the mean; this tells us that while life expectancy is 78 years, there is a high likelihood that one could die much earlier or much later.
Graph 4: This is a bimodal distribution – it tells us that we have higher mortality at earlier years (approximately 58 years) and later years (approximately 98 years). While the mean of this distribution is 78 years, most people, in raw numbers do not perish at 78 – they either die much earlier or much later.
For us to effectively compare means, we have to make an assumption that the underlying distributions are normally distributed – i.e., graphs 1 and 3 above. Moreover, we also must know the standard deviation of the distribution to make effective comparisons.
Incidentally, if we suspect the impact of getting healthcare to more people is a healthier population, I would suggest that simply looking at the distributions would be sufficient – in this case, we would hypothesize that countries with uniform, available healthcare might look like Graph 1, while countries with less available healthcare options might have distributions like those in graphs 2-4. In this case, the variation is the more important metric, not the mean.
3. Exchange rate and geographic pricing difference
The data shown have been converted to U.S. dollars for the sake of comparability. This could say more about the valuation of the U.S. dollar than the state of healthcare; moreover, it is not uncommon for pricing practices among healthcare organizations to differ geographically based on market pressures, governmental controls, and other factors.
4. Selection bias
The World Health Organization has data on many countries throughout the world; the example I have shown focuses on four countries. The extent to which these are not representative could have a dramatic impact on the confidence we place not only in the analysis, but the strength of our recommendations moving forward. Worse, if the examples have been hand-picked to reinforce our preferred argument, then we cannot make a scientific conclusion from the data provided.
5. Overly simplistic models
Even if we assume that the core business question is clearly articulated, the model shown is too simple to capture the nuances of such a complex issue. For example, the data do not allow us to control for any variables that may have an adverse impact on per capita spend levels. Several have been articulated in the debate - for example, the extent to which limits are placed on lawsuits, which impacts the cost of business as it relates to necessary tests and physician compensation (for malpractice insurance premiums) as well as the nature of the competitive environment (which places downward pressure on prices).
Simple models, while attractive, usually have limited utility; in addition, the simpler the model, the more defined and specific the underlying business question needs to be.
These are just five things to consider (I’m sure there are more – please let me know if you have other ideas). In my next blog (the last in this series), we will explore what we can take away from this example as we focus on the topic of customer loyalty management analysis.
Mark Ratekin
Sr. Vice President, Consulting Services & Resource Management
As good analysts, we should scrutinize any data to ensure that it is not only appropriately used (and that any limitations to its use are noted and considered when making recommendations), but also that it adequately addresses the core business question. Of course, for this to occur, we must have a well-defined, properly framed business question.
I set out a challenge for our readers to consider what limitations existed in the data and how this might constrain what conclusions we could make about the linkage between healthcare spending and life expectancy. In this blog, I’ll review five factors that could impact how we interpret the data.
1. What is the underlying question we are trying to answer?
The first issue that creates confusion (and, potentially, misinterpretation of the data) relates to the underlying question at hand – is the goal in the healthcare debate to increase life expectancy overall, to minimize the expense of healthcare, to get coverage to more people, or something else? Of course, like the business questions we face daily, there is not a single answer; I would suggest, however, that pundits are trying to answer too many questions with a single data set.
2. Is the outcome metric the right metric?
The second issue relates to the core outcome metric being employed – mean life expectancy. While longer life expectancy is good, it does not tell us the whole story; to know that, we must understand the underlying distribution of the data. Consider the following data distributions:
Graph 1 Graph 2
Graph 3 Graph 4
The shapes of these distributions clearly suggest differences at play in the data; they do, however, have one thing in common – their mean value (78 in this case).
Let’s explore what these might mean from the perspective of life expectancy – how would we interpret each of these?
Graph 1: The variation around the mean is fairly constrained; this suggests that most people will live within a fairly tight range around 78 years.
Graph 2: The data skew toward a lower life expectancy (roughly 58 years). What this means is while the average life expectancy is 78 years, many people (about one-third) will not live past 63 years; on the other hand, there are a great volume of people that live much longer lives, which effectively increases the average.
Graph 3: This is a normally-distributed set of data with wide variation around the mean; this tells us that while life expectancy is 78 years, there is a high likelihood that one could die much earlier or much later.
Graph 4: This is a bimodal distribution – it tells us that we have higher mortality at earlier years (approximately 58 years) and later years (approximately 98 years). While the mean of this distribution is 78 years, most people, in raw numbers do not perish at 78 – they either die much earlier or much later.
For us to effectively compare means, we have to make an assumption that the underlying distributions are normally distributed – i.e., graphs 1 and 3 above. Moreover, we also must know the standard deviation of the distribution to make effective comparisons.
Incidentally, if we suspect the impact of getting healthcare to more people is a healthier population, I would suggest that simply looking at the distributions would be sufficient – in this case, we would hypothesize that countries with uniform, available healthcare might look like Graph 1, while countries with less available healthcare options might have distributions like those in graphs 2-4. In this case, the variation is the more important metric, not the mean.
3. Exchange rate and geographic pricing difference
The data shown have been converted to U.S. dollars for the sake of comparability. This could say more about the valuation of the U.S. dollar than the state of healthcare; moreover, it is not uncommon for pricing practices among healthcare organizations to differ geographically based on market pressures, governmental controls, and other factors.
4. Selection bias
The World Health Organization has data on many countries throughout the world; the example I have shown focuses on four countries. The extent to which these are not representative could have a dramatic impact on the confidence we place not only in the analysis, but the strength of our recommendations moving forward. Worse, if the examples have been hand-picked to reinforce our preferred argument, then we cannot make a scientific conclusion from the data provided.
5. Overly simplistic models
Even if we assume that the core business question is clearly articulated, the model shown is too simple to capture the nuances of such a complex issue. For example, the data do not allow us to control for any variables that may have an adverse impact on per capita spend levels. Several have been articulated in the debate - for example, the extent to which limits are placed on lawsuits, which impacts the cost of business as it relates to necessary tests and physician compensation (for malpractice insurance premiums) as well as the nature of the competitive environment (which places downward pressure on prices).
Simple models, while attractive, usually have limited utility; in addition, the simpler the model, the more defined and specific the underlying business question needs to be.
These are just five things to consider (I’m sure there are more – please let me know if you have other ideas). In my next blog (the last in this series), we will explore what we can take away from this example as we focus on the topic of customer loyalty management analysis.
Mark Ratekin
Sr. Vice President, Consulting Services & Resource Management
As the debate around healthcare reform in the U.S. rages on, we are inundated with data from either side of the political spectrum that strive to reinforce the major points of the plans under proposal. The media is even doing comparative analysis; on CNN, Lou Dobbs has been running series of segments comparing the current U.S. healthcare system to other systems around the world. These segments (one of which can be viewed here), are full of stats and metrics, including patient satisfaction levels.
One of the common practices in these comparison is to compare spending levels (in absolute spend or percentage of country GDP) with average life expectancy. The theory is that greater spend levels should yield longer life expectancy. Consider the following data:
Location | Per capita total expenditure on health at average exchange rate (US$) | Total expenditure on health as % of GDP | Life expectancy at birth (years) both sexes |
United States of America | $6,714 | 15.3% | 78 |
Canada | $3,912 | 10.0% | 81 |
Japan | $2,690 | 7.9% | 83 |
Sweden | $3,870 | 8.9% | 81 |
These data do not bear out that theory – the country in these data with the longest life expectancy, Japan, spends the least on healthcare. What’s going on?
Before we jump to the conclusion that healthcare is too expensive relative to the outcome, I would suggest that we take a critical look at the data to see what, beyond public policy, could lead us to the data we see here.
What do you think? If you have thoughts, please leave a comment. One caveat, though – it is not my goal to open up a political discussion. Please consider only the data and possible sources of bias or error that could impact our interpretation.
In my next blog, I’ll look at five factors that may impact the interpretation.
Mark Ratekin
Sr. Vice President, Consulting Services and Resource Management
There are more ways than ever before to efficiently analyze information. All of this readily available information can be overwhelming, but using it wisely can provide robust, actionable guidance. The following paragraphs attempt to provide some guidance on navigating all of this available information.
· First and foremost, know the question(s) you are trying to answer. Sample design, survey design and how the data is analyzed, all should tie-back to the question.
· Understand your sample plan- know to whom your findings do and do not generalize.
· Do your homework before designing the survey. This will help to ensure that important measures are not excluded.
o There are a myriad of available statistics to help assess the goodness of the model and variables measured. The findings from these techniques can be used to guide survey design moving forward.
· Use all pieces of available information to answer the question- survey results- both closed and open-ended measures, behavioral data, voice of the customer, competitive data and information, historical data, news and industry information, etc.
o Using all available information helps to provide a robust answer to the question, but keep this in-mind- some findings may appear to be inconsistent. These apparent inconsistencies can be another valuable source of information (assuming no errors have been made).
· Understand the level of data being used- is it being analyzed at an individual customer level, or is the data being used at an aggregate level (e.g., by account, by call center, by region, etc.).
o Post-data collection is also the time to re-evaluate sample representativeness- are there customer non-response issues, how do the sample distributions align with the population distributions, etc.
· Use the most appropriate statistics to effectively and efficiently answer questions.
o Rather than just comparing performance scores, statistical testing can be used to narrow down where statistically meaningful differences exist between scores.
o Multivariate analytic techniques can further help to direct focus on the measures that most effectively distinguish or help in understanding customer attitudes and behaviors.
o Do not assume that if you do not see patterns in the data, that none exist. Sometimes the failure to see patterns in the data is a result of not appropriately segmenting the data. As a follow-up to customer segmentation, profiling techniques can be applied for further understanding and description of customer segments.
· Answering the question often leads to new questions being asked. A single program cannot/should not be designed to answer all questions.
o Tracking the questions and answers over time, and analyzing how these things have evolved, can again, lead to further insights. Review this process periodically to adapt to the ever-changing business world.
Amy Heleine
Director, Marketing Sciences
A man leaves work on a stormy afternoon, only to find that he is unable to catch a cab. This is because all the cabs have been taken by the many other commuters who don’t want to walk home in the rain, right? One group of researchers argues that while this DOES reflect increased demand, it is also the result of decreased supply, and exemplifies the potential de-motivating effects that goals can have. The researchers found that cab drivers would set daily wage goals. On sunny days, one would have to drive a full shift in order to reach those goals. But with the increased demand on rainy days, the drivers would find success much more quickly. As drivers reached their daily wage targets, they would end their shifts early, calling that day a success. This would leave an increased number of soggy commuters to battle over a ride home (Camerer, Babcock, Loewenstein, & Thaler, 1997).
This represents a challenge that companies face as they approach and eventually achieve the goals they have set for themselves. Authors Ordóñez, Schweitzer, Galinsky, and Bazerman point out that people tend to view goals as “ceilings” rather than “floors.” They suggest that an employee who has reached his assigned goal is one who, in his own eyes, has earned the right to slack off for the remainder of the associated timeframe. A natural remedy might be to set a higher goal once the original is achieved. This IS a recommended approach when dealing with short term goals designed to incrementally make progress toward a lofty long-term result. In other cases though, this approach can get you into trouble.
Consider, for example, any metric that is represented as a percentage. You may start by setting a goal of 85%. Once that is achieved, you might choose to raise the bar to 90%. But as performance improves, further progress becomes more and more of a challenge. There may even come a point when the next targeted score is actually impossible. Remember that there is often a limit (in this case, 100%).
This brings me to what is undoubtedly the most important component of goal-based management: the desired business outcome (discussed in Parts 2, 3, and 5 of this series). Never lose sight of WHY you set a goal in the first place!
If there is no more room for improvement in the target metric and the desired outcome has not yet been realized, you’ll need to determine why that is. Maybe the indicators of that ultimate outcome have shifted, or perhaps the originally-targeted metric is simply one of multiple contributors to that final indicator of success. If, on the other hand, achieving your goal has already led you to that business outcome, then congratulations are in order. It’s time to kick back and let your business run itself.
Cortney Lantry
Director, Marketing Sciences
Resources
Ordonez, Lisa D.; Maurice E. Schweitzer; Adam D. Galinsky; and Max H. Bazerman (2009). "Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Settings" Harvard Business School Working Paper.
Camerer, C., Babcock, L., Loewenstein, G., & Thaler, R. (1997). Labor Supply of New York
City Cabdrivers: One Day at a Time. The Quarterly Journal of Economics, 112(2), 407- 441.
After emerging from bankruptcy during the 1990s, an airline focused efforts on reducing costs. To motivate employees toward the goal, the company offered rewards to pilots who effectively reduced fuel costs. The plan worked. Pilots did reduce fuel costs…by adjusting air conditioning and reducing flight speed. The changes resulted in late and uncomfortable flights, so customers switched to other airlines.
Desired Business Outcome: Reduced total costs.
Selected Metric: Fuel costs (presumably to be reduced by a specified percentage).
Action: Reward pilots who minimize fuel costs on their flights.
Oversight: Fuel cost is tied to aspects of the business that affect customer perceptions.
The airline offers a real example of one of the dangers of linking employee compensation to company goals. The approach will likely be effective, but could result in a case of, “be careful what you wish for…”
At the opposite end of the spectrum is an automobile manufacturer that failed to gain employee buy-in on the company goal. Executives observed low productivity (38.98 hours to assemble a vehicle) and a high error rate (179 problems per 100 vehicles produced). They communicated the improvement goals, showing that they were vital to success. But employees did not strive for the goals because they feared that improved productivity would result in job loss.
Desired Business Outcome: Improved performance.
Selected Metrics: Assembly time, Problems per vehicle.
Action: Communicate the importance of the goals throughout the company.
Oversight: Employees saw no benefit (and in fact suspected job loss) associated with achievement of the goal.
This company learned from the experience and went on to develop a more successful system. They dispelled the fears about job security and introduced incentive to work harder. Employees who achieved their productivity goals would be redeployed to more intellectually challenging activities within the organization. The result was a reduction in error rate (from 179 to 134 problems per 100 vehicles) and quicker assembly (from 38.98 to 24.44 hours per vehicle).
Incentives are often necessary to motivate employees to strive for the goals that will benefit the company. Reward systems must be implemented carefully, though. If they are not thoroughly researched and closely monitored, the incentives can result in success at high cost, or even in unethical behavior. If they are not enticing enough, the investment in the research and development of the goals may yield little to no return.
A well-developed goal-based management system sets realistic targets and gives employees a reason to help the company succeed. With that strong foundation, the next issue to face is the happy dilemma of what to do after you have achieved your goal!
Cortney Lantry
Director, Marketing Sciences
Resources
Sisk, Michael (2003). Are the Wrong Metrics Driving Your Strategy?. Harvard Management Update, November 2003.
Ordonez, Lisa D.; Maurice E. Schweitzer; Adam D. Galinsky; and Max H. Bazerman (2009). "Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Settings" Harvard Business School Working Paper
In order to be considered among the best in the industry, your company would like to be able to claim a 95% success rate in responding to technical service requests. You survey customers after each technical support transaction is completed, asking whether the request was resolved in a timely manner. You monitor results on a quarterly basis to learn what percentage of customers report that ‘Yes,’ the request was resolved in a timely manner.*
Assumptions: Your approach makes the following assumptions. Move on ONLY if you determine that these are appropriate:
· Maintaining a 95% success rate among technical support requests will qualify your company as best in the industry.
· A ‘Yes’ response to “Was your request resolved in a timely manner?” indicates technical support success.
· Employees can directly impact this measure.
Observations:
· Scores are FAR from the ultimate target of 95%.
· Scores are not consistent from one quarter to the next.
· Scores are cyclical. There is a drop in performance at each second quarter.
Setting a goal:
· Metric – Since there is great fluctuation in scores over time, your quarterly metric may need adjustment. Consider the following:
o Monitor quarterly evaluations, but aim for greater CONSISTENCY before trying for improvement.
o If quarterly consistency is not realistic given the business cycle, aim for semi-annual or annual targets that increase steadily until 95% is eventually reached.
o Consider whether another metric might be more stable, more comparable to what other top companies are monitoring, more indicative of technical support success, etc .
· Target value – 95% is not realistic at this time. Many alternatives exist, including:
o Computing the average annual score and setting an initial annual target that represents a statistically significant improvement.
o Targeting a quarterly score that is a statistically significant improvement over the score for that same quarter in the previous year.
Some questions that naturally follow the goal-setting process include:
· how best to gain employee support, and
· how to not only achieve the goal but also build upon that success.
The next part of the Goals series will explore these topics, including the heavily debated topic of linking goals to employee compensation.
* This survey question, bivariate scale, and census approach (surveying after EVERY transaction) are used here for simplicity of example. This is not a recommended design in all cases.
Cortney Lantry
Director, Marketing Sciences
The business world has introduced me to the BHAG - Big Hairy Audacious Goal. We do love our acronyms, but we don’t need to keep them all. Your BHAG may sound impressive, but it won’t go far in gaining employee buy-in or making steady, sustainable progress.
Send BHAG packing……….!
To begin a more effective approach to setting business goals, a definition is in order. Within this context, a “goal” is a specific target for a measurable metric that can be monitored over time. A strong goal is different from (but directly related to) a desired business outcome. Ask yourself, “WHY should the company strive to reach this goal?” The answer to that question is the desired business outcome. If you do not have an answer to that question, STOP RIGHT NOW because every aspect of successful goal-based management depends upon a proven link between the goal metric and your desired outcome.
| Metric | Specific Value | Business Outcome |
1. | Total fixed costs | 10% reduction | Increased net income |
2. | Customer complaint rate | 2.7% | Higher overall satisfaction with transactions |
Simplicity is important here. Consider #1 above. It might be tempting to jump straight to a goal of $X in net income, but consider all the factors involved in achieving that goal. Your “goal” is not a wish. It is something that you will have to manage, so do your best to make it manageable. If your business requires a lofty goal, break it down into components that you can work with.
Once you have identified your desired outcome and the associated metric(s), some questions remain:
· What should be the target value of the chosen metric?
· What timeframe should be set for achieving the goal?
· How can I communicate the importance of this goal to gain employee support?
· Do monetary incentives work?
· What happens when the goal is achieved?
Stay tuned….
Cortney Lantry
Director, Marketing SciencesImagine that your business has hit a plateau. You are able to attract new customers, but you have experienced very little growth because you are losing customers just as quickly.
Your Business Objective: Achieve sustained net revenue growth.
A Related Business Problem: Customer churn rate is high.
Your Business Goal: Reduce churn rate.
You now have a plan, and you need buy-in from your employees. So you go one step further.
Management Strategy: Offer bonus pay for all employees who effectively maintain customers.
But you have overlooked a key metric. Only 25% of your customers are highly profitable. Most of your revenue originates from 25% of your customer base.
Consequences: You waste resources on keeping customers that offer little to no ROI, and miss an opportunity to attract new customers who fit the highly profitable profile.
In this example, the mistake is in simply selecting a metric that happens to be readily available, without considering whether achieving this goal will necessarily lead to sustained net revenue growth.
Other traps to avoid when setting goals:
· choosing the wrong metric
· setting too many goals
· limiting focus to financial/stock metrics
· allowing an inappropriate timeframe for reaching the goal
· setting goals that are too challenging or not challenging enough
· limiting goals to improvement upon past performance rather than customizing them to the future direction of the company
· making erroneous assumptions about the data used to track progress
Here you have a “Don’t” list. Upcoming entries will discuss the “Do” list, as well as another component of the example above – incentivizing by linking goals to compensation.
Cortney Lantry
Director, Marketing Sciences
Resources
Sisk, Michael (2003). Are the Wrong Metrics Driving Your Strategy?. Harvard Management Update, November 2003.
Researchers ask a group of people to watch a video in which two teams are passing basketballs. One team wears white shirts, while the other wears dark shirts. The viewers are assigned the task of counting the passes among the white-shirt team. In concentrating on this objective, the viewers unconsciously block out the players wearing dark shirts. They become so focused on the players wearing white shirts that they don’t notice when a man wearing a black gorilla suit enters the scene, pounds his chest, and walks off screen (Simons & Chabris, 1999).
Measuring company performance leads quite naturally to the impulse to set a goal. Goal-setting is a favorite tool used by companies in an effort to motivate employees and to improve business performance. While this strategy certainly has its merits, it is not a blanket business solution. Extensive consideration must go into determining the suitability of a goal, identifying a metric, and determining effective short and long-term targets. In the example above, the participants focused so narrowly on their goal that they became blind to surrounding circumstances. If such tunnel-vision can occur in a controlled lab setting, imagine the potential oversights in such a complex environment as the business world!
Noted benefits of goal-based management:
· Goal-based management presents unambiguous expectations.
· It allows for objective evaluation of performance.
· Attention and resources are focused on a common objective.
· Goals serve as motivation to increase efforts.
Associated risks:
· Such narrow focus can inhibit creativity, innovation, and learning.
· It can increase competition, thereby decreasing cooperation.
· Pressure to meet a goal sometimes results in unethical employee behavior.
· The goal may be achieved at the cost of other essential components for success.
This is the first post within a series that explores the potential benefits, as well as the dangers, in relying upon goals in business. The series offers opportunity for discussion on how to effectively set and manage goals (using both statistical and practical applications), should they be deemed appropriate for your business.
Cortney Lantry
Director, Marketing Sciences
Resources
Simons, D. J., & Chabris , C. F. (1999). Gorillas in our midst: sustained inattentional blindness for dynamic events. Perception, 28(9), 1059-1074.
Ordonez, Lisa D.; Maurice E. Schweitzer; Adam D. Galinsky; and Max H. Bazerman (2009). "Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Settings" Harvard Business School Working Paper.
Recently, I read an article about mistakes that are commonly made in Marketing Research. Since so many business decisions are made from the results of research, I felt it was important to be aware of some of the most typical pitfalls and how we might be able to avoid them.
1. Halo Effect, or "the common tendency to make specific inferences on the basis of an overall impression." The idea is that if a firm is performing well, people tend to think of it as being customer focused, well-managed, and a whole host of other positive things; however, the reverse happens when the firm starts performing poorly.
· Suggestion: When testing concepts such as “customer focused”, the author suggests not actually asking this question but a series of more fact-like questions such as "Are customers involved in the product design process?" or "How frequently is customer satisfaction measured?"
· Caution: It is important to measure customer attitudes in the most efficient way possible, which typically involves asking one question directly instead of asking two to three questions that get at the idea.
2. Assuming causality from a simple correlation. With a correlation, there is no way of knowing if X causes Y or if Y causes X, just that there is a relationship between them.
· Suggestion: Remember when interpreting correlations that the only thing they tell us is how strongly related the two variables are. Any claims of how one variable impacts another should come from regression results.
3. Assuming that all variables that could possibly affect the dependent have been controlled.
· Suggestion: Pay attention to the R-square values in the analysis. These will let you know how much of the variance of the dependent variable has been captured by the variables we have included in the model.
4. Thinking that following a given set of steps will ensure high performance. The truth is, the existence of competition makes performance relative, not absolute. A company can do all the right things, but if a competitor does them faster and better, typically the competitor's performance scores will rise while the company's will fall.
· Suggestion: Collect and analyze competitive information alongside data about your company. This helps get a feel for the true landscape, and is particularly important in highly competitive fields.
Let this be food for thought as you reflect on your own research programs. Think carefully about how you are interpreting the findings and be sure not to fall into any of these common traps!
Jessica Gregory
Reference: Rosenzweig, Phil. 2008. "Common error in marketing research -- and how to fix them." Marketing Research, Fall2008, Vol. 20 Issue 3 p:6-12.
Last week, I discussed some of the issues we need to be aware of when considering the use of net scores. Before moving forward, let’s be clear – there are no bad metrics as long as they meet a number of criteria:
Ultimately, we would advise our clients that the best metrics offer actionability while communicating a core central message in a motivational way.
Mark Ratekin
Sr. Vice President, Resource Management and Consulting Services
- The metrics need to resonate within your organization – They need to be understood, accepted, and embedded in the culture.
- The metrics must create an impetus for action – Score-keeping in and of itself is neither strategic nor imperative; for metrics to be valuable, they must facilitate action in a way that provides guidance on where, how, and how much to change.
- The metrics must link to business outcomes – This seems obvious, but it is a step that many companies overlook. It is tempting to use conventional wisdom as our proof point; in other words, if it makes sense that loyal customers (or satisfied customers, or Promoters, or committed customers, etc.) buy more, then that should be proof enough. Not so – to borrow a quote from Ronald Reagan, we should trust but verify. Making certain our metrics link to hard business outcomes ensures that the action we take today will yield the results we are looking for tomorrow.
- Top 2 (or Bottom 2) box scores are intuitively easy to understand; for example, an 80% top two box on an Excellent-Poor scale means that 80% of customers score us as either Excellent or Very Good. Moreover, we immediately know that 20% of the customers rate us Good, Fair or Poor. Of course, this presents a challenge in that we don’t know how the 20% are dispersed across Good, Fair and Poor – this is meaningful, as you would likely interpret a customer rating you “Poor” differently than one rating you “Good.”
- Means offer the advantage of considering all scale points, but are harder to interpret - for example, if I score a 4.2 mean on an Excellent – Poor scale, how do I interpret this? In addition, while the mean considers all scale points, it does not address the distribution – in other words, even when we know the mean, we don’t know anything about how the scores are dispersed across the scale (we need the standard deviation for that).
- When Top 2 performance gets to the point that it’s too large to be meaningful, many clients migrate to a top-box score metric. “Excessive” top 2 box scores are also an indicator that we may want to revisit the model to ensure that we are using the best metrics to predict future behavior.
Ultimately, we would advise our clients that the best metrics offer actionability while communicating a core central message in a motivational way.
Mark Ratekin
Sr. Vice President, Resource Management and Consulting Services
I had the opportunity recently to discuss the issue of net metrics with a client; given the popularity of the topic of net scores (thanks in large part to the Net Promoter metric), I thought it would be beneficial to share our thoughts on the topic in this blog. In this first installment, I’ll talk about some of the methodological issues you should consider when thinking about net scores.
Net scores are composite scores that are defined by taking the difference between two scores; this can be from two different questions (for example, the Top 2 score of Quality minus the Top 2 score of Value) or from within a single question (for example, the Top 2 score minus the Bottom 2 score).
Net scores are touted to be effective due to their ease of understanding; assuming this is true, we give up a couple of key statistical elements that we believe most clients would find troubling from a target setting and/or actionability perspective:
-If we decompose the net score into its component parts, we see a much different picture:
Given these scores, do we really think both groups are making the same progress? In this example, Entity A is making progress in reducing the number of Bottom 2 scores, while Entity B is growing the Top 2 Box scores. The point is, we need to understand how each element is moving relative to the other to truly assess who is improving and who is declining.
The whole premise of using net scores is that they are easier to understand – there is a school of thought that says that any metric that requires some level of calculation is, by definition, more difficult to understand. The scenario cited above reinforces this relative to net scores; the same could be said of indices as well. As with most metrics, you’ll need to consider your culture and the organization’s ability to assimilate and utilize any metric – after all, measurement for the sake of measurement is neither strategic nor a wise use of the firm’s time and resources. Creating a bias for action is the key.
In my next entry, I’ll review some common ways our clients use metrics and how these drive action. In the meantime, I’d be interested in your thoughts – what metrics do you use in your organization? Do they resonate with your co-workers? Do they prompt action?
Mark Ratekin
Sr. Vice President, Resource Management and Consulting Services
Net scores are composite scores that are defined by taking the difference between two scores; this can be from two different questions (for example, the Top 2 score of Quality minus the Top 2 score of Value) or from within a single question (for example, the Top 2 score minus the Bottom 2 score).
Net scores are touted to be effective due to their ease of understanding; assuming this is true, we give up a couple of key statistical elements that we believe most clients would find troubling from a target setting and/or actionability perspective:
- The margin of error around net scores balloons as a result of the need to account for variance in both portions of the metric. The result can be a metric with a margin of error so wide as to be meaningless. For example, if our net score is 70 with a margin of error of +/- 20, this means our actual results fall within a range of 50 to 90. The only way to counteract this is to dramatically increase the sample size. Increases in sample can, of course, have cost implications, particularly depending on the method of data collection employed. This limitation is particularly critical, given the desire to set performance targets – a wide margin of error can prove to be not only difficult, but also de-motivating.
- Net scores are just that – net. As a result, when we see movement, we cannot discern what portion of the score is moving. Consider the following scenario
-Two groups, Entity A and Entity B, each move their net score from 40 in period one to 60 in period two. From a net score perspective, they are doing equally great.
-If we decompose the net score into its component parts, we see a much different picture:
Wave 1 | Wave 2 | |||||
Group | Top 2 | Bottom 2 | Net | Top 2 | Bottom 2 | Net |
Entity A | 65% | 25% | 40% | 63% | 3% | 60% |
Entity B | 50% | 10% | 40% | 77% | 17% | 60% |
Given these scores, do we really think both groups are making the same progress? In this example, Entity A is making progress in reducing the number of Bottom 2 scores, while Entity B is growing the Top 2 Box scores. The point is, we need to understand how each element is moving relative to the other to truly assess who is improving and who is declining.
- The nature of a net score is, by design, an aggregate level indicator. It has no utility in helping us to profile individual client relationships (unless we combine, as NPS does, individual score points into categories like Promoter and Detractor).
- Finally – and perhaps most problematic from an actionability perspective – the utility of net scores (when using a single metric) is limited to the aggregate level. Unless we arrive at categorical definitions (again, like the NPS approach), the data by itself can have constrained value.
The whole premise of using net scores is that they are easier to understand – there is a school of thought that says that any metric that requires some level of calculation is, by definition, more difficult to understand. The scenario cited above reinforces this relative to net scores; the same could be said of indices as well. As with most metrics, you’ll need to consider your culture and the organization’s ability to assimilate and utilize any metric – after all, measurement for the sake of measurement is neither strategic nor a wise use of the firm’s time and resources. Creating a bias for action is the key.
In my next entry, I’ll review some common ways our clients use metrics and how these drive action. In the meantime, I’d be interested in your thoughts – what metrics do you use in your organization? Do they resonate with your co-workers? Do they prompt action?
Mark Ratekin
Sr. Vice President, Resource Management and Consulting Services
In an age of growing social media, it is important to understand how communication about a product affects buyers’ decision making. A growing body of research focuses on exploring this connection and has produced several relevant findings:
· The decision to adopt a brand is affected by the information obtained from the brand's users and by information from the competition's users. It has been shown that within-brand communication has a larger effect on buyers’ adoption decision than cross-brand communication does.
· There is usually an advantage for the first entrant into the market, as their customers generate word-of-mouth about the brand. Hence, they grow at a larger rate than the competitor, who enters later and is only benefitting from the weaker effect of cross-brand communication generated from the first entrant's customers.
· Later entrants to the market do often experience a faster initial growth than the first entrant. This is because the first entrant did not have the benefit of cross-brand communication.
Perhaps the key takeaway from the research, however, is that communication can be a bigger influence on the buying decision than several other factors, including:
· Price Differences, specifically reductions in price made by the first entrant when its competitor enters the market.
· Network Effects or special perks associated with adopting a particular brand (such as pricing schemes that offer lower rates for talking to other customers within that network).
· Improved technology that enables higher quality and lower cost, typically used by the later entrant.
· Control of infrastructure by the first entrant who has had time to gain control over the resources before the entrance of the competition.
The reminder for all of us is that customers are a strategic asset. It is important to remember to treat them as such; their word of mouth counts more than you might realize!
Jessica Gregory
Statistical Analyst
Reference: Libai, Barak, Eitan Muller, & Renana Peres. 2009. "The Role of Within-Brand and Cross-Brand Communications in Competitive Growth," Journal of Marketing, May 2009, Volume 73 Issue 3, p19-34.
My name is Olivia Taurel, and I am an intern at Walker Information for the summer. At the end of my internship, I will be going back to Bloomington for my senior year at Indiana University, where I study Sociology and International Studies. I take a particular interest in sociology courses surrounding gender issues, hence my excitement about this blog…
A recent study on gender differences in customer loyalty revealed some interesting and noteworthy results (Bijmolt, et al., 2009). The key takeaways from the article, “Are Women More Loyal Customers than Men? Gender Differences in Loyalty to Firms and Individual Service Providers,” are:
· Gender affects customer loyalty—women are more loyal to individual employees or service providers, and men are more loyal to groups of people or companies.
·Female customer defection should be anticipated when an employee leaves a company, therefore requiring readily available measures to accommodate for this possibility.
·Advertising strategies should correspond to the preferences of the gender of the customer base.
These results offer some managerial implications:
·Companies targeting women should encourage their employees to develop relationships with their customers, while companies with a predominantly male customer base can rely more on anonymous relationships with their customers.
·The focus of advertising strategies can and should be dependent upon the gender of the customer:
·Male-targeted advertising should focus on group themes.
·Female-targeted advertising should focus more on personal relationships.
While this research provides some interesting insight into the effects of gender on customer loyalty, it is one of the first to study the issue; further research needs to be conducted before generalization within B2C and B2B relationships can be established. This study does, however, bring some valuable information to the table that can assist with segmentation in a broader context. More information on this topic will follow.
Olivia Taurel
Intern
Businesses rely on numbers. Some use more than others, but at the heart of every company is a steady stream of numbers. There are two types of numbers (aka, metrics) we rely on in business: 1) those that measure past/current performance, and 2) those that forecast future performance. The past/current metrics are often difficult to gather and precisely measure. Those of us specializing in customer consulting and analyzing customer feedback have to measure and craft strategies based on customer perceptions and attitudes, which are notoriously difficult to measure and track.
While companies rely heavily on current metrics to run and evaluate their business, we have become extremely reliant on forecasted metrics for our strategic planning and decision-making. And if it's difficult to get accurate current metrics, how much harder is it to get accurate future metrics? However, executives, business leaders, and investors everywhere want, even demand, forecasts of key operating metrics with little regard for the certain inaccuracies of the forecast.
The extremely intriguing book, The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb, which took me all last summer to read, argues that humanity is awful at understanding the causal mechanisms that produced current or past events and is, therefore, incapable of accurately understanding the likelihood of future events except in a few small and somewhat meaningless domains.
He also discusses the impact of a forecasted number on human decision-making. Citing experimental results from a couple of psychologists, he illustrates a mental mechanism called anchoring. In the experiment, subjects were asked to spin a wheel with numbers on it and then focus on the number, which they knew was completely random. The subjects were then asked to estimate the number of African countries in the UN. Those who had spun a low number estimated a significantly lower number of countries than those who had randomly spun a higher number. In another experiment, subjects were asked for the last four digits of their social security number and then asked to estimate the number of dentists in Manhattan. Their estimates were strongly correlated to last four digits of their SSN.
In other words, providing someone with a number, regardless of how strong the caveats are concerning its accuracy, immediately changes how they think. In essence, people lower their anxiety about future uncertainty by anchoring to a number that is purported to "predict" the future, and it doesn't matter how many caveats you put around it. If people's thinking is so easily affected by numbers they know to be random, think of the impact of a number produced by fancy algorithms that decision-makers have some reason to believe in.
Obviously, a lot more can, and has been, written on this topic, but I want to provide a few ideas that can help us approach the process of forecasting, and the communication of those forecasts, in a more honest way.
- Acknowledge the potential inaccuracy of forecasts. They say acceptance is the first step to recovery.
- Forecast ranges instead of single numbers. Acknowledge the variability of the outcome. I'm currently preparing for a trip to Sonoma, CA. If I only considered the daily high temperature or the average daily temperature when packing clothes, I would only take shorts and t-shirts. However, I know to consider the daily range of temperatures and to pack a light coat for the cool evenings. However, in my home state of Indiana it is currently ranging from incredibly hot and humid to just mildly hot and humid, so I don't even know where my light coat is! All this to say, you should give your audience a range of likely outcomes based on the variability of the outcome, not just one number.
- Actually look at the accuracy of past forecasts of the metric. It's amazing how seldom people look back at the accuracy of past forecasts before running the forecast for the next year - often using the same forecasting method even though the last forecast was horribly inaccurate. Why? Sometimes the last forecast has already been forgotten. It was used to craft a strategy and then never looked at again. Other times it is because we dismiss the inaccuracy as the result of unexpected events or causes that are "outside of the model" (exogenous variables for you quants). Well, if those events happened last time, why do we think other "unexpected" events won't happen again?
- Always be mindful of the potential for a large, random event that completely changes the conditions of the forecast. Even if a metric has been relatively stable in the past, don't assume it is immune to the effect of a highly improbable event.
- Don't produce a forecasted number that you know to be inaccurate just because "it's your job." This is at best unethical, and at worst criminal. There is ample recent evidence that unthinking, or unethical, forecasters can cause substantially more damage to society than criminals.
So be responsible when you forecast. Remember that other people's livelihood often relies on the forecasts you produce and the impact those forecasts have on decision-making.
As a parting side note, you'll be interested to know that the accuracy of weather forecasters far outperforms economic forecasters' accuracy. Think of that the next time you're watching the rain out your window with a fully packed picnic sitting next to you.
Troy Powell, Ph.D.Vice President, Statistical Solutions
Walker Information
Powered by Compendium Blogware