This tutorial is going to teach you about the difference between a Type I error and a Type II error in a hypothesis test. You will specifically focus on:
When you think about a hypothesis test as a decision making tool, it's possible that you could be making some errors. Suppose we're in a clinical trial for a new drug. There are two possibilities for whether the drug is, in fact, an effective drug or not. Either it is or it's not.
When you use a hypothesis test as a decision-making tool, you might make a different decision. There are two possibilities for the decision you arrive at. You could fail to reject the null hypothesis of the drug being not effective or you could reject it in favor of the alternative, the drug is effective. One of those two will be our conclusion.
But there's only one thing that's actually true and fact. Suppose these are the four different possibilities. Two of them are OK. These are correct decisions.
If the drug was effective, you should reject the null hypothesis and decide that the drug is effective. If the drug isn't effective, you should fail to reject the null hypothesis and decide that the drug isn't as effective as it would have needed to be to reject it. But if you look below, the drug is, in fact, not effective but you decide that it is.
That is called a Type I error. The drug is not effective but you rejected the null hypothesis anyway. Based on your data, you thought that you had enough evidence to reject the null hypothesis. But, in fact, the drug is not effective. And see on the chart below, the drug was effective but the data that we got didn't make it clear enough, and so you failed to reject the null hypothesis. This is another incorrect decision. It's called a Type II error. So, there are two different types of errors that we can make.
Type I Error
An error that occurs when a true null hypothesis is rejected.
Type II Error
An error that occurs when a false null hypothesis is not rejected.
What are the consequences of each of those? Think back to a Type I error versus a Type II error. A Type I error would have a consequence of you approve the drug and allow the public to have it, even though it's not effective. But you're also unleashing all the potential negative side effects that this drug might have. There's really no upside here and some negative consequences.
In a Type II error, you would not allow the drug to go to market because you think it's not effective when, in fact, it is. You would deny an effective drug that you didn't know was effective, based on our data. Your data made you think it wasn't, but it is, in fact, an effective drug to the public who might need it. This is another negative consequence. These errors always have negative consequences.
Which one are you more easily able to sleep at night with? In this case propbably a Type II error. You have a hard time dealing with this idea of unleashing something that might hurt people just because you think it might be effective. Typically, you need some really hard evidence. If there's not hard evidence, we would deny the drug.
In the criminal justice system, juries are told to presume that someone is innocent until proven guilty. Meaning the null hypothesis is that the suspect is innocent, and the prosecution has to prove its case.
What would a Type I and Type II error look like in this context?
A Type I error would be that the person is innocent but they're convicted anyway. A Type II error would be that the person is guilty but the result of the trial is that they're acquitted. Obviously, both of these are problematic. But the criminal justice system in America puts a lot of safeguards in place to make sure that a Type I error doesn't happen very often.
In fact, the criminal justice system allows a Type II error to happen fairly frequently in order to reduce a Type I error. Because you think a Type I error is absolutely the worst thing you can do in this particular case. It's not always this way. Sometimes a Type II error is worse. It depends on the situation, and so, you have to analyze each situation to determine which one is a worse mistake to make.
When you talk about a hypothesis test as a decision-making tool, you might be making an error in our judgment. It's not that you made a mistake, but the result that you choose might not match what is really the case.
A Type I error is when the null hypothesis is rejected when it's true. In fact, that is the alpha level. That is the significance level that you create in a hypothesis test. A Type II error is when the null hypothesis is not rejected. In reality, it's actually not the case, it's false. So, it's false but you didn't reject it.
The severity of these errors depends on the context. In both the examples that you did, a Type I error was worse. But there is conceivably some scenarios where a Type II error might be worse.
Source: This work adapted from Sophia Author Jonathan Osters.
In a hypothesis test, when the null hypothesis is rejected when it is in fact, true.
In a hypothesis test, when the null hypothesis is not rejected when it is, in fact, false.