Response Bias

Video Transcription

This tutorial is going to talk to you about response bias. Now, response bias is when people's answers are influenced by something. So, a nice way to think about sampling is with a pot of soup analogy. When you get a representative sample, that's like getting a little taste of everything that's in the soup.

But things can go wrong, where you don't get the right taste of the soup. Response bias can occur if the wording of the question is such that it's unclear to the respondent, or if a respondent is uncomfortable due to sensitive or personal nature of the questions, or if the respondent feels like the questioner is implying that the question has a quote unquote "correct" response. That's also called social desirability bias.

So, for instance, here's an example. On April 20, 1993, the New York Times published an article on a survey conducted by the Roper Organization on behalf of the Jewish American Community about the soon-to-be opened Holocaust Museum in Washington, DC, right here.

The newspaper reported that 22%, an astounding number of adults surveyed, expressed some doubt as to whether the Holocaust had actually occurred. The actual question that was presented to people was, "Does it seem possible, or does it seem impossible to you, that the Nazi extermination of the Jews never happened?"

Now, on its surface, this seems to be a fairly straightforward question, but there was a big problem with it, and it caused response bias. The problem is that the question contains a double negative. "Does it seem impossible to you that it never happened?" Saying it's impossible that it never happened is the same as saying that you're certain that it did happen. It just doesn't sound that way when it's read in the question.

The good thing is that, one year later, the question was revised, and it became clearer. The new question stated, "Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?" With this new, clearer question, the question clearly distinguishes between what the two options are-- "does it seem possible," or "do you feel certain?"

And now, with the two options clearly defined, less than 2% of individuals were unsure as to whether it was real or not. And this gives a more accurate interpretation of what the American public feels. So, unclear questions can lead to an inaccurate representation due to response bias.

The other scenario in which this can occur is when people will answer a question because they are either ashamed, or they think that there's a right answer that someone's fishing for. So, in a survey about drug use, many people will say they've never used it, even if they have. And even if there's no consequence, and even if the survey is anonymous, they'll still say they've never used when, in fact, they have.

There are certain topics that are particularly sensitive and might make a person want to lie. Criminal history-- they might say they don't have one, even if they do; sexual behavior, because it's of a highly sensitive and personal nature; racial prejudice, because there's an implied right answer. People don't want to say that they're racially prejudiced.

And income-- people will report it as being higher than it actually is if they're of low income status, or even possibly more surprisingly, people will report it as lower than it really is if they're of a very high income status. A lot of people don't want to be very showy about their wealth, and so they'll try and come up with a more-- in their eyes-- reasonable number.

Now, how does this affect what we think about the population? How does this affect the "soup?" Well, it's like taking a sample of the soup and only tasting the things that you want to taste. Maybe you don't like beans, and so you just sort of ignore the fact that they're there. Well, you don't get the overall flavor of what's supposed to happen.

Same thing with response bias. It doesn't give you the right overall interpretation of what this is supposed to be like.

And so, to recap, response bias occurs one of two ways. Either a respondent doesn't understand the question and so gives an answer that he wasn't intending, or because the respondent wants to give a supposedly correct answer to the questioner. Both of these can be inaccurate representations about what actually is the truth about the population. Response bias is a tough thing to get rid of, especially when it deals with the wording of the questions and it's unintentional. The terms we used were response bias.

Good luck, and we'll see you next time.