Last week, the following question was posed- Can customer performance be trended when methodological changes are implemented? Yes, a regression-based conversion provides a bridge for trending when a scale or text based methodological change is made. This week, the focus will be on scale/text changes; next week, the focus will be on survey mode changes.
A data conversion analysis uses a simple regression equation to predict the new metric based on responses to the original metric. The end result is a trending bridge- a score of x% (or x mean score) on the original metric, corresponds to a score of x% (or x mean score) on the new metric. While the trending solution is simple, the design needs careful consideration in-order to ensure a sound conversion.
· Manipulate one thing at a time-
o If moving from a 10-point scale to a 5-point scale, question text should be kept the same for the two measures, only the scales should differ.
o If changing question text, the scales should be kept the same for the two measures, only the question text should differ.
· Each respondent should receive the old metric and the new metric, as respondents need to answer both questions for the regression model.
· Randomize the order in which respondents receive the old and new metric; this helps control for order bias.
· Understand your measures, as different types of questions/scales perform differently-
o The old and new metric should be asked for each different type of question (e.g., attitudinal measures versus image measures versus experiential measures) and each different type of scale (e.g., excellent-to-poor versus satisfaction versus agreement).
· A conversion requires a substantial base size- at least 300 respondents. A large base size is required in-order to ensure stability of the mean-conversion regression model. In addition, the large base size is particularly needed if a conversion will made from a mean to a percentage, as this requires drawing repeated sub-sets of the data in-order to build a stable regression model.
Thinking back to the question posed in the first blog on this topic- can customer performance be trended when methodological changes are implemented- the simple answer to the question was, “yes.” However, for the answer to be simple, the design on which the answer is based must be sound. Thoughtful planning of the design will help to ensure that this is the case. Next week, we will discuss survey mode changes…
Director, Marketing Sciences