Response rate is a topic that seems to be growing in importance throughout literature and survey methodology. In evaluating the response rate literature, it is important to consider how response rate is defined as well as what the response rate is telling you about your study.
Defining Response Rate: How do I calculate response rate?
Response rate is a term that can have a variety of definitions simply depending on who you are talking to. Looking at how response rate was calculated in 16 articles, there are at least 6 different calculations that were used. CASRO (Council of American Survey Research Organization) has defined response rate as the “ratio of the number of interviews to the number of eligible units in the sample.” But even this definition leaves room for various interpretations.
Number of Interviews:
· Does this only include individuals that completed the entire survey?
· Do they have to complete at least 50% of the survey?
· How do you handle respondents that only answered a few questions in the survey?
· How do you handle people that simply clicked into the survey?
Number of Eligible Units:
· Does this exclude undeliverable invites?
· Does this only include the people who clicked into the survey?
· Is this simply the number of invites that were sent?
As you can see, depending on the calculation you choose you could get a wide range of results for your study’s response rate. The way you calculate response rate will also impact calculations of item non-response and drop-off rates.
The most common calculation that I have seen throughout the literature is:
# of people who answered at least one question in the survey
# of invites sent - # of undeliverable invites
Evaluating Data Quality: Should Response Rate be the only thing I look at?
Usually after the response rate is calculated, we want to then assess that number to see how our study stacks up to similar studies. It is important to make sure you understand which calculation method is being used by the studies that you use as benchmarks. Often the response rate is given without any of the details about how it was calculated.
Response rate is sometimes considered a way to measure the quality of your study. However, research has shown that larger response rates do not equal better data quality or more accurate results. So response rate should be one of multiple elements you consider. Some other elements to consider are sample quality, sample representativeness, non-response bias, and amount of missing data (item non-response, drop-off rates, etc).
While response rate should not be the only metric used to assess survey quality, it is an important consideration when collecting data. In the coming weeks, we will be examining how various survey methodology choices impact response rate, such as mode and survey design. We will also consider ways to increase your response rate.