This tutorial is going to discuss response bias by focusing specifically on:
Response bias is when people's answers are influenced. A nice way to think about sampling is with a pot of soup analogy. When you get a representative sample, that's like getting a little taste of everything in the soup.
But things can go wrong and you don't get the right taste of the soup. Response bias can occur if the wording of the question is unclear to the respondent, or if a respondent is uncomfortable due to sensitive or personal nature of the questions, or if the respondent feels like the questioner is implying that the question has a quote unquote "correct" response. That's also called social desirability bias.
On April 20, 1993, the New York Times published an article on a survey conducted by the Roper Organization on behalf of the Jewish American Community about the soon-to-be opened Holocaust Museum in Washington, DC, right here.
The newspaper reported that 22%, an astounding number of adults surveyed, expressed some doubt as to whether the Holocaust had actually occurred. The actual question that was presented to people was:
"Does it seem possible, or does it seem impossible to you, that the Nazi extermination of the Jews never happened?"
This seems to be a fairly straightforward question, but there was a big problem with it, and it caused response bias. The problem is that the question contained a double negative.
The good thing is that, one year later, the question was revised, and it became clearer. The new question stated:
"Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?"
With this new, clearer question, the question clearly distinguishes between what the two options are-- "does it seem possible," or "do you feel certain?"
With the two options clearly defined, less than 2% of individuals were unsure as to whether it was real or not. This gives a more accurate interpretation of what the American public feels. So, unclear questions can lead to an inaccurate representation due to response bias.
The other scenario in which this can occur is when people will answer a question because they are either ashamed, or they think that there's a right answer that someone's fishing for.
A survey about drug use may exhibit many people saying they've never used it, even if they have. And even if there's no consequence, and even if the survey is anonymous, they'll still say they've never used when, in fact, they have.
There are certain topics that are particularly sensitive and might make a person want to lie:
How does this affect what we think about the population? How does this affect the "soup?" I's like taking a sample of the soup and only tasting the things that you want to taste. Maybe you don't like beans, and so you just sort of ignore the fact that they're there. You don't get the overall flavor of what's supposed to happen.
Same thing with response bias. It doesn't give you the right overall interpretation of what this is supposed to be like.
Response bias occurs one of two ways. Either a respondent doesn't understand the question and so gives an answer that he wasn't intending, or because the respondent wants to give a supposedly correct answer to the questioner.
Both of these can be inaccurate representations about what actually is the truth about the population. Response bias is a tough thing to get rid of, especially when it deals with the wording of the questions and it's unintentional.
Source: This work is adapted from Sophia author jonathan Osters.