Type I & Type II errors

Many students, many texts…

Many students trying to understand Type I and Type II errors encounter a table similar to the one below. It is a common table found in many introductory, and intermediate statistics texts. Some conjuring in the universe mandate that they commit the table to memory. Many fail in their attempt to do so. It gets better. There are eye witness accounts of applied statisticians across disciplines and fields, who once committed the table to memory but fail to correctly regurgitate the details.

Type I and Type II errors based on the NHST framework

	\(H_0\) is true.	\(H_0\) is false.
Do not reject \(H_0\) (\(p \geq \alpha\))	We made the correct decision to not reject \(H_0\). We concluded there is no sign of an effect.	Type II error rate (\(p = \beta\)). Do not reject \(H_0\) when it is false. False negative: We missed the effect.
Reject \(H_0\) (\(p < \alpha\))	Type I error rate (\(p = \alpha\)). Reject \(H_0\) when it is true. False positive: We wrongly claimed an effect.	Correctly reject \(H_0\). We found an effect.

If you happen to be the exception, I salute you! May you continue to dwell in the presence of the statistics gods. If, like me, you have ever been fooled by Esu, a simple anecdote may save you because this anecdote couldn’t care less for your ability to memorise. By simply relating a courtroom scenario to the statistics context, you may be better equipped to apply the knowledge for years to come. I’ll take that again. This post uses a simple courtroom scenario to bring home the concept of Type I and Type II errors. The aim is to help students gain a better intuition of the topic. However, use of the courtroom context for this purpose is no novel idea. One may find similar illustrations by simply conducting a web search. But when did that ever stop me from having my take on things?

Type I error (\(\alpha\))

Imagine this. You are in a courtroom where someone called Unlucky is on trial for murder. Using the Null Hypothesis Significance Testing (NHST) framework, the judge is relying on the following hypothesis statements to arrive at a decision:

\[\begin{align} H_0: \text{Unlucky is innocent.} \\ H_1: \text{Unlucky is guilty.} \end{align}\]

At the end of the trial, the judge decided that Unlucky is guilty. Unlucky is then sentenced to 30 years imprisonment with hard labour. Note, the judgment was based on evidence (sample data). Some evidence suggested that Unlucky committed the crime. More specifically, the police found Unlucky’s fingerprints on the victim.¹

The logic of NHST states that if, based on evidence (sample data), we can reject \(H_0\), then \(H_0\) must be true. This is because \(H_0\) and \(H_1\) are mutually exclusive – they cannot occur at the same time. If one is true, the other must be false. If Unlucky is truly guilty, they cannot be innocent and vice versa.

However, in the true state of the world (population), Unlucky is innocent. In other words, the judge rejected \(H_0\) when it was true. This is known as a false positive or Type I error (\(\alpha\)). Limited evidence (sample data) led the judge to believe that Unlucky was guilty whereas in reality, Unlucky was innocent. In statistics, the probability of making Type I error is \(\alpha\). \(\alpha\) is predetermined – chosen by the researcher or by the research context. For instance, a 95% confidence level implies \(\alpha = 1 – 0.95 = 0.05\). This means the probability of making Type I error (i.e., declaring that Unlucky is guilty when in reality, they are innocent) is \(0.05\).

Type II error (\(\beta\))

Imagine that time around, someone else called Lucky is on trial for murder. Using the NHST framework, the judge has similar hypothesis statements as before:

\[\begin{align} H_0: \text{Lucky is innocent.} \\ H_1: \text{Lucky is guilty.} \end{align}\]

The defense presented evidence (sample data) indicating that Lucky had an alibi; therefore, Lucky could not have committed the crime. The judge informed the court that to convict Lucky, the prosecution had to provide stronger evidence to indicate Lucky actually committed the crime and to counter the defense’s argument (smaller \(\alpha\); \(\alpha < 0.05\)). This can be thought of as presenting evidence (larger \(n\))². In court, the prosecution could not present stronger evidence that would have compelled the judge to conduct a more rigorous test of \(H_0\) (e.g., testing with smaller \(\alpha\)).

At the end of the trial, the judge decided that Lucky was innocent.Lucky was therefore discharged and acquitted. As before, the judgment was based on existing evidence (sample data). That is, there was evidence (sample data) suggesting that Lucky had an alibi; therefore, could not have committed the crime.³

The logic of NHST states that if, based on evidence (sample data), we cannot reject \(H_0\), then either \(H_0\) or \(H_1\) may be plausible.

However, in the true state of the world (the population), Lucky committed the crime. The judge ought to have rejected \(H_0\) because it was a false statement. In other words, the judge failed to reject \(H_0\) when it was false. This is known as a false negative or Type II error (\(\beta\)). The evidence presented by the defence (sample data) led the judge to believe that Lucky was innocent whereas in reality, Lucky was guilty. The probability of making Type II error is known as Beta (\(\beta\)). \(\beta\) is determined by a number of factors such as, \(\alpha\), sample size (\(n\)), the research design, and the true effect size in the population, which has different measures depending on context (e.g., Cohen’s \(d\)).

Conclusion

Imagine you were the judge. Hopefully, you can imagine that you could easily have made either Type I or Type II error. In real world courtrooms, these errors are made from time to time, which explains why some innocent people are convicted of crimes they did not commit, end up serving prison sentences, and having criminal records.

Also, it is important to note that \(\alpha\) and \(\beta\) are inevitable trade-offs. In statistics and in research, a workable solution that minimizes \(\alpha\) and \(\beta\) is to make use of large sample sizes (\(n\)) whenever possible. Holding other factors constant, for a stated Type I error rate (\(\alpha\)), a large sample size (\(n\)) implies a smaller Type II error rate (\(\beta\)).

As a final point of reflection, which of these errors do you think is more common among statisticians?

Footnotes

1. Evidence, just like samples, may or may not reflect the true state of the world (population).

2. In applied settings, one way to achieve this is to produce different results based on a different but equal sample size (\(n\)) obtained from the same population. Another approach is produce different results based on a larger (\(n\)) obtained from the same population. Obviously, the second approach is preferred due to larger \(n\). Producing similar results based on the same sample size \(n\) taken from the same population would only strengthen evidence in favour of Type I error - assuming of course that the research context allowed for a high probability of this error.

3. There is a trade-off between Type I error (\(\alpha\)) and Type II error (\(\beta\)). To reduce the probability of Type I error, we can use smaller values of \(\alpha\) (e.g., \(\alpha\) = 0.01) to test \(H_0\) compared to the coventional \(\alpha = 0.05\) in the social sciences. This will be analogous to the judge asking for stronger evidence in order to avoid convicting a truly innocent person (Type I error). However, by asking for stronger evidence (e.g., \(\alpha = 0.01\)), the judge will inevitably increase the probability of discharging and acquitting a truly guilty person – that is, increasing the probability of making Type II error (\(\beta\)). As you may already be able to tell, common law countries such as the U.K, Australia, Malaysia, and Canada, prefer a justice system with a higher probability of making Type II error compared to Type I error. Such countries would rather let a potentially guilty person go free due to lack of evidence than convict a truly innocent person.