Biostatistics for MS2 Students

Part 4: Types of Error

• Previous
• Contents
• Next

Remember the framework for statistical hypothesis testing:

In this system, two things can "go wrong":

1. The null hypothesis is really true (so our alternative hypothesis is false); however we just happen to get a p-value which is below the threshold \(\alpha\) (usually 0.05). In this scenario we incorrectly reject the null hypothesis and incorrectly conclude that our alternative hypothesis of interest is true. This is called a false positive, or a Type I Error.
2. The null hypothesis is really false (so our alternative hypothesis is true); however our experiment or study fails to yield a p-value which is below the threshold \(\alpha\). Usually this means we get a p-value greater than 0.05. In this scenario we fail to reject the null hypothesis, even though it is false, and fail to conclude that our (correct) alternative hypothesis is true. This is called a false negative, or a Type II Error

It is important to understand that the word "Error" here is not being used to mean that something was wrong with the study or experiment. Type I and Type II Errors can, and will, occur even in perfectly-designed studies. The p-value is computed from the data from our sample, which is a random selection of the population. Because of the sampling being random, we will sometimes get a sample which is sufficiently misrepresentative of the population to give a result of the experiment or study which is inconsistent with what is really happening.

This situation is commonly summarized in the following table. It is important to understand that, in this table, we never know what is "really true"; i.e. we never really know which column we are in:

			Real actual status
		p-value compared to \(\alpha\) (\(\alpha\) is usually 0.05)	Null hypothesis is false (our hypothesis is true)	Null hypothesis is true (our hypothesis is false)
Result of experiment or study	Reject null hypothesis (positive result)	\(p\lt\alpha\)	True positive	False positive Type I Error
	Do not reject null hypothesis (negative result)	\(p\gt\alpha\)	False negative Type II Error	True negative

Controlling the error rates \(\alpha\) and \(\beta\)

The false positive rate is the chance of getting a false positive if the null hypothesis is true. In the table above, it's the proportion of false results in the right-hand column. This is exactly the value \(\alpha\); the threshold we use for statistical significance.

We clearly would like all error rates to be as small as possible. However, if we choose to make \(\alpha\) smaller, with everything else kept equal we will increase the chances of a false negative. In theory, choosing the false positive rate (the threshold for statistical significance) should be considered as a trade-off with the desired false negative rate, and should depend on subjective judgements about the severity of the consequences of a false positive or false negative result. In practice, this is difficult to do, and typically researchers just use a "standard value" of 0.05 for \(\alpha\).

The false negative rate is the chance of getting a false negative if the null hypothesis is false. In the table above, it's the proportion of false results in the left column. The false negative rate is denoted \(\beta\), and is affected by the following:

• The false positive rate. The lower the value of \(\alpha\), the higher \(\beta\) will be.
• The strength of the association we are trying to establish in the alternative hypothesis. If there is a strong association; i.e. if the explanatory variable has a large effect on the outcome, then the false negative rate will be lower than if there is a weak association (the explanatory variable has a small effect on the outcome).
• For outcomes measured by continuous variables, the amount of variability in the outcome affects the false negative rate. If there is a large amount of variability, the false negative rate will be higher then if there is a small amount of variability.
• For outcomes measured by discrete variables (such as whether or not a patient has a disease, or a mortality outcome for a disease), the false negative rate is affected by the chances of the outcomes. If all outcomes are equally likely, \(\beta\) will be lower than if one or more of the outcomes are highly unlikely.
• The sample size. The larger the sample size, the smaller \(\beta\) will be.

Notice that most of the above are beyond our control; the strength of the association, the amount of variability, and the chances of each individual outcome are simply intrinsic properties of the outcomes we are studying. While we can control the false positive rate \(\alpha\), we can only reduce \(\beta\) by increasing \(\alpha\), which is undesirable. The one remaining parameter under our control is the sample size.

This means that in theory we can choose an arbitrarily small false positive rate \(\alpha\), and then choose a sample size large enough that we achieve a false negative rate as small as we like. There are two reasons this might not work in practice:

1. A large sample size may be impractical, or we may not have sufficient resources (in terms of money or time) to perform a study with a large sample size.
2. The extra effort expended in conducting an experiment or study may not be worth the return we get in reduction in \(\beta\). Some level of risk of a false negative is acceptable.

Sample size and power calculations

Sample size calculations involve computing the sample size required to achieve a suitably small false negative rate at a chosen false positive rate. These calculations should be performed at the time an experiment or study is being designed. Funding agencies, such as the National Institutes of Health, which provide financial support for experiments and studies typically require that sample size calculations are included as part of an application for funding. They want to see both that the study has a reasonable chance of success (assuming the alternative hypothesis being tested is true), and also that the study is not unnecessarily consuming resources.

Statistical Power

The statistical power is the chance an experiment or study finds a postive result (i.e. that the p-value is below the threshold: \(p\lt\alpha\)) assuming that the alternative hypothesis is really true (i.e. the null hypothesis is false). This is simply the opposite result to a false negative, so it the statistical power is \(1-\beta\). It is fairly standard to design experiments (in particular, to choose sample sizes) so that the statistical power is 80%: \(1-\beta=0.8\).

Meta-Analyses

One way to increase the sample size of a study is to combine the results of previous studies into a single study. This kind of analysis is called a meta-analysis. Note that while this increases the sample size, since each individual study will have been performed slightly differently, there is a possibility that the additional variability introduced will nullify the effects of the increased sample size. Specialized statistical analyses design for meta-analyses are used to mitigate this effect.

Summary

Did I understand this?

1. In a grant application, the potential investigators outline a sample size calculation in which they state that \(n\) is chosen to give a statistical power of 80% at a statistical significance level of 0.05. Which of the following is true?

   1. \(\alpha=0.05\) and \(\beta=0.8\)
   2. \(\alpha=0.95\) and \(\beta=0.8\)
   3. \(\alpha=0.05\) and \(\beta=0.2\)
   4. \(\alpha=0.95\) and \(\beta=0.2\)
   5. \(\alpha=0.95\) and \(\beta=0.02\)
   Incorrect

   • \(\alpha\) is the false positive rate, which is the level of statistical significance.
   • \(\beta\) is the false negative rate
   • Statistical power is the chances of a postive result, if the null hypothesis is true. It is the opposite of a false negative, so it has value \(1-\beta\)

   Correct!

   • \(\alpha\) is the false positive rate, which is the level of statistical significance.
   • \(\beta\) is the false negative rate
   • Statistical power is the chances of a postive result, if the null hypothesis is true. It is the opposite of a false negative, so it has value \(1-\beta\)

   In this case, the statistical significance level is 0.05, so \(\alpha=0.05\), and the power is 0.8, so \(1-\beta=0.8\), and \(\beta=0.2\).

2. Suppose the investigators in the proposal described in question 1 wanted a statistical power of 95% instead of 80%. What could they do to achieve this?

   1. It is impossible to have a statistical power of 95% at a statistical significance level of 0.05.
   2. Perform the sample size calculation with a power of 95%, obtaining a smaller sample size.
   3. Perform the sample size calculation with a power of 95%, obtaining a larger sample size.
   4. Increase the strength of the association of the explanatory variable and the outcome of the study.
   5. Reduce the amount of variability in the outcome variable in the study.
   Incorrect

   Increasing the statistical power means decreasing the false negative rate. This can be achieved by increasing the sample size. Any arbitrarily small false negative rate can be achieved, but may require impractical sample sizes.

   Incorrect

   The strength of the relationship and the amount of variability in the outcome of the study are properties of the actual population data (i.e. they are real, fixed, values). These cannot be altered by the investigators.

   Correct!

   To increase the statistical power (i.e. decrease the false negative rate \(\beta\)), the sample size needs to be increased. A new sample size calculation can give the new \(n\) required.