Nonresponse Bias

Video Transcription

This tutorial is going to teach you about nonresponse bias. Now, bias we've already talked about as being a bad thing, and we're going to talk about how nonresponse, or a lack of response, from people you've selected is going to affect the ability to draw conclusions from your sample.

A nice way to think of sampling is with what we call a "pot of soup" analogy. So, we want a representative sample, which means we don't need to drink the entire pot of soup in order to figure out what's in it. We just need the right taste.

So, it would be like selecting all of the ingredients from the soup in a single tasting, but certain things can go wrong with the taste test that can affect what we think is in the soup because, in real life, we don't really know what the population looks like. We don't know what's in the soup. All we get is the taste, and if we don't get the right taste, we're going to leave something out and not know exactly what's in the soup.

So, let's go back to the sampling world here for a minute. Nonresponse means that someone selected for the sample either can't be contacted or is unwilling to participate. So, suppose someone gets a call, and they say, "Hi, you've been selected to take a sample." They say no thanks and hang up. This is problematic.

Nonresponse in and of itself-- it's not the end of the world. It's fine. It happens. It's an inevitability that you either will get people that are uncooperative and don't want to take your survey, or answer your questions, or be part of your experiment, or it's inevitable that you just won't be able to contact certain people.

The problem comes in when the opinions of the people left out-- the people that weren't able to be contacted or refused to participate-- differ substantially from the people that were in the sample, and that's problematic. That's called nonresponse bias because you're not getting an accurate cross-section of opinions.

So, let's go back to the analogy of the soup for a second. How does that affect the taste test? Well, we don't get an accurate flavor profile from our taste of the soup because some of the ingredients have been left out. Some of the opinions of the people that we wanted to get are left out.

So here's an example. A workplace wishes to survey 200 of its 1,000 employees about their workload and their stress level, so they put 200 surveys in the workers' mailboxes. Now, what might happen is that the people who have the biggest workloads might get left out of the sample because they don't get around to checking their mailboxes because they're already so busy. Or, even if they do get around to checking their mailbox, maybe they don't fill out the survey, or don't return it, because they're so busy.

What effect might that have? Well, of the 200 that the workplace actually gets back, maybe the ones that it gets back say that the workload level is not that high. The only problem is the people with the lower workloads are the only people who turned them in, because they had the time to take it. And the people with the higher workloads didn't have the time to take it. The company might think the workload level is lower than it really is.

Take a look at these different ways of conducting a survey, or a poll, or a sample. Which of these methods, mail, telephone, or face-to-face, do you think has the highest nonresponse rate? The answer is the mail. People will either throw it away, or forget to fill it out, or maybe they'll fill it out and then forget to mail it back. This is kind of problematic because when the United States takes its census of everyone in the country, it does it by mail. And so, sometimes they have to do follow-ups.

The nonresponse rate is easy to calculate. You just subtract the number that you got back from the number that you mailed out, and that's your nonresponse rate. Say you mailed out 100, and you only got 80 back. Well, that's 20 out of 100, or 20% nonresponse rate. In samples with high rates of nonresponse, follow-ups typically are needed.

So, supposing you started with a mailing, you might need to follow up by calling them at home. And if you can't reach them by calling them at home, you might need to follow up by coming directly to their house. And sometimes, even then, even when they are contacted, someone will refuse to participate. Follow-ups like this might be more necessary in some areas of the country than others because different areas of the country have different rates of nonresponse.

And so, to recap, nonresponse bias occurs when people who are selected for the sample can't participate, either because you can't find them, or because they're actively refusing. And the biggest problem is that if you have high rates of nonresponse, it might give you an inaccurate representation of what's going on with your population. You won't be able to use your sample to draw an inference about your population.

So, we talked about nonresponse, which is just the act of being unable to participate in the survey, and nonresponse bias, which is the problems that arise as a result.

Good luck, and we'll see you next time.