Think back to a previous lesson where you learned about the experimental method. Specifically think back to the fourth step, the one where you test the predictions regarding how variables interact and record data.
In statistics, there are populations and samples. Experiments use samples in an effort to learn more about a population. Two typical problems that occur when conducting experiments can impact the results of that experiment. One problem is what is referred to as bias, and the other problem is associated with not accounting for all variables, specifically those referred to as extraneous variables.
Bias is the result of something in the experiment causing measurements that are not representative of the population. Bias causes the parameters of a population to be misrepresented as the parameter is typically over- or underestimated. Because of bias, data collected from an experiment cannot be applied to a greater group of things. That the data is not reliable.
Say a group of friends or coworkers decides to see who can lose the most weight in a month. In a case like this, bias would exist if the scale is not accurately measuring the weight of those who use it. If the scale hasn’t been calibrated and it’s off, that could have an impact on the results of the weight loss contest. While conducting the experiment, any error resulting from the way the experiment is designed is known as bias.
Unless this bias is addressed, the error will still exist if the experiment is repeated. People keep getting on the same scale over and over again. It’s going to be wrong every single time. Therein lies the bias.
Even if an experiment does not have any bias, the results can still be impacted by random errors. In the case of the weight loss contest, a random error could be the case of somebody simply misreading the scale and writing down the wrong weight.
The goal of the experimental method is to determine whether or not there is a cause-and-effect relationship between variables. In addition to any bias it has encountered, all variables that might affect the response variable being studied must be accounted for.
Those variables that are not accounted for are called extraneous variables. Extraneous variables can cause a misinterpretation of the cause-and-effect relationship that may exist between two variables.
Lars goes to the doctor’s office, and complains about certain symptoms that might be indicative of a particular condition. The doctor prescribes a drug and asks Lars to come back in a couple weeks. At his follow-up appointment, Lars’ symptoms are gone.
While the prescribed drug may have worked, it’s possible that the condition was an allergic reaction to another medicine. If Lars stopped taking that initial medicine, well, this might be a case of an extraneous variable. It wasn’t the medicine the doctor prescribed that caused the symptoms go away. It was related to something else.
When conducting an experiment, there are two types of bias that require special attention: selection bias and participation bias. Selection bias can occur when the random assignment of individuals or things to the groups in an experiment is not done at all or is not even possible.
Random assignment is done so you don’t over or under represent a specific portion of the population. You randomly pick people or observations to try to avoid having all the observations with the same characteristics.
A good example of selection bias would be a telephone poll where a poller or an experimenter decides to call people and ask them questions about a particular issue. The experimenter picks the landline numbers based on public record. The experimenter is specifically under representing all people who might have a telephone by missing people that don't use a landline.
The selection bias would be only choosing people that have a particular type of a telephone number, a landline, which under represents the population. It’s going to introduce bias into the results here. Observations that are left out of the study like this may cause a study to not represent the larger group of individuals being studied.
Participation bias is a special type of selection bias that can occur when participants have a choice whether to participate in an experiment. Those who choose to participate may have very different characteristics from those who choose not to, and that can influence the results of an experiment.
Say your shopping receipt has a code at the bottom and the salesperson tells you, “If you type in this website and answer the survey questions, you’ll be entered to win a $5,000 gift card.” The possibility of winning the gift card may prompt somebody to participate in the survey, and participation that is incentivized in this way creates "participation bias". Participation in the survey (the experiment) is entirely up to the individual.
There are multiple steps in the experimental methods process. In Steps 2, 6, and 7 of the process, whether we are coming up with a hypothesis, drawing a conclusion about an experiment, or simply trying to revise our hypothesis, taking into consideration bias and its effect on the experiment leads to stronger conclusions. Multiple variables that can impact the response variable being studied in an experiment. Typically, errors in an experiment can be the result of variables that are not considered by the researcher. These are called extraneous variables.
When selection or participation bias occurs, the groups being studied might have features that are impacted by an extraneous variable that was not considered. Random assignment reduces the effects of extraneous variables, since fewer participants with the same characteristics are assigned to a specific group.
When there is no random assignment and many participants have the same characteristics, the extraneous variable may impact the group being tested in the same way, and alter the results of the data.
To illustrate this idea, we’ll use an SAT prep course. The extraneous variable is how much time the students spend studying outside of class. It is very possible an experimenter could draw a false conclusion based on the fact that they simply took the class to begin with, and did not consider how much time they spend studying outside of the class.
When reading statistical results, it is important to look for such extraneous variables since they can be responsible for misleading or even incorrect results. Therefore, you need to take them into consideration when working in the experimental methods process, much like you take bias into consideration.
Source: This work is adapted from Sophia author Dan Laub.
Variables that are not controlled in the experimental design.
An error caused by the experimental design creating a situation where the values related to parameters of a population will consistently be over or under estimated.