Biostatistics for MS2 Students

Part 5: Confidence Intervals

In any study or experiment, we collect data from a sample. We want to use these data to make inferences about a more general population. It is fairly easy to do calculations on the data from the sample, and use these as estimates of the population. However, we usually need some way to understand how accurate these estimates are.

For example, consider the AZT study from Cooper et al. (1993) that we saw in the module on Hypothesis Testing. In this study, the investigators were testing whether the drug Zidovudine (AZT) affected the propensity of patients with asymptomatic HIV infection to progress to symptomatic disease. A sample of 475 patients were treated with AZT and 461 with placebo for a period of three years. Results are shown in the following table:

Disease status
SymptomaticNot symptomaticTotal
TreatmentAZT76399475
Placebo129332461
Total205731936
From these data, we can compute the odds of disease progression. The odds are calculated as the number of those who progressed to disease, divided by the number who did not. So we have
TreatmentOdds of disease progression
AZT\[\frac{76}{399}=0.190\]
Placebo\[\frac{129}{332}=0.389\]
Finally, we can compute the odds ratio for the sample of patients in the study: \[\text{Odds Ratio (OR)}=\frac{0.190}{0.389}=0.49\] This indicates that the odds of progressing to disease for the AZT treatment group in the sample are 0.49 times the odds of progressing to disease for the placebo group in the sample. This is our best estimate of the odds ratio for the population. However, we would like some measure of how accurate this estimate is.

One way to express the accuracy of this estimate is with a confidence interval. For any level of confidence we choose, we can compute an interval, which we believe contains the actual odds ratio for the population as a whole, with that degree of confidence. It is traditional to compute confidence intervals for a 95% level of confidence. We could do this for these data, and we would state:
The odds ratio for progression to disease for the AZT group relative to the control group was 0.49, with a 95% confidence interval of 0.351 to 0.682.
This means that our best estimate of the odds ratio, given our data, is 0.49, and we are 95% confident that the range 0.351 to 0.682 contains the true odds ratio.

It is important to state here that 95% is simply a value we chose to compute an illustrative range. We could equally well have chosen to compute and report a 90% confidence interval, or a 99% confidence interval, but 95% is the traditional choice.

The measure of accuracy of the odds ratio estimate is not the confidence level, but the interval that we computed. Here the interval spans from 0.351 to 0.682, so we have a high degree of confidence, in particular, that the odds ratio is less than 0.682.

Relationship between Confidence Intervals and Hypothesis Testing

In almost any statistical hypothesis test, we can state the null hypothesis as a statement that a statistic of interest is equal to a particular value (occasionally called the null value). For example, in the AZT study, the null hypothesis is that AZT does not affect the chances of progression to disease. We could state this equivalently as the odds of progression to disease for the AZT group being the same as the odds of progression to disease for the placebo group. Since the ratio of two equal values (equal odds in this case) is 1, the null hypothesis here can be stated as "the odds ratio is 1".

In statistical hypothesis testing, we choose a threshold for our p-value, \(\alpha\), (usually we choose \(\alpha=0.95\)). We then collect data and compute the p-value. If the p-vale is less than \(\alpha\), we conclude the null hypothesis is false. The p-value is the probability of obtaining data as extreme as that observed, under the assumption that the null hypothesis is true.

This is strongly connected to the idea of a confidence interval. Here our null hypothesis is that the odds ratio is equal to 1. Asking if there is less than a 0.05 chance of observing the odds ratio we observed in our study, assuming the true odds ratio is 1, is logically equivalent to asking if we are 95% confident that the actual odds ratio is not equal to 1.

In other words,

The p-value for the null hypothesis that the odds ratio is equal to 1 will be less than 0.05 precisely when the 95% confidence interval for the odds ratio does not contain 1.

Suppose we conduct a study like the one above, but in which there is no difference between the treatment and control groups: in other words, we'll assume the null hypothesis is true and that the odds ratio is 1.

If we were to do this same study many times, we would have a different sample of patients every time, so we would have different estimates of the odds ratio and different 95% confidence intervals every time. On average, 95% of those confidence intervals would contain the "true" odds ratio, which, since we're assuming the null hypothesis is true, is 1.

On the other hand, the p-value is the probability of observing an odds ratio at least as extreme as the one observed in the data, assuming the null hypothesis is true. If we repeat the study many times, and the null hypothesis is true, then on average 5% of the studies will give a p-value less than 0.05, and these will be the 5% with the most extreme odds ratios (the ones furthest from the true value of 1).

Since the 95% confidence intervals are chosen so that 95% percent of them include the value 1, the 5% of studies which have odds ratios furthest from 1 are precisely those whose confidence intervals do not include 1.

Did I understand this?

  1. In a recently published study, Kim et al. studied the relationship between moderate lifetime alcohol intake and amyloid-beta deposition in the human brain. Since moderate lifetime drinking has previously been shown to be protective against Alzheimers Disease (AD), and since amyloid-beta deposits are a known risk factor for AD, they wanted to know if a history of moderate drinking affected the level of amyloid-beta deposition in the brain. They recruited patients from three categories of lifetime drinking history (no drinking, moderate drinking, and heavy drinking) and measured amyloid-beta deposition in the brain.

    The investigators computed the odds of deleterious amyloid-beta deposits for the group with a lifetime history of moderate drinking, and similarly computed the odds of deleterious amyloid-beta deposits for the group with a lifetime history of abstinence (no drinking). The odds ratio was 0.341 with a 95% confidence interval 0.163 to 0.714.

    The investigators similarly computed the odds ratio of deleterious amyloid-beta deposits for a group of current moderate drinkers versus a group who currently do not drink. The odds ratio for this comparison was 0.503, with a 95% confidence interval of 0.241 to 1.004.

    What conclusions can we draw from these results?

    1. Neither a lifetime history of moderate drinking, or a current moderate drinking status, affect amyloid-beta deposit levels
    2. A lifetime history of moderate drinking decreases the level of amyloid-beta deposits, but a current moderate drinking status has no effect.
    3. A current moderate drinking decreases the level of amyloid-beta deposits, but a lifetime history of moderate drinking has no effect.
    4. A current moderate drinking decreases the level of amyloid-beta deposits, and no definitive conclusion can be drawn about the effect of a lifetime history of moderate drinking.
    5. A lifetime history of moderate drinking decreases the level of amyloid-beta deposits, but a no definitive conclusion can be drawn about the effect of a current moderate drinking status.
    Incorrect
    Since the 95% confidence interval for the odds ratio for the group with a lifetime history of moderate drinking does not include 1, the p-value for that comparison is less than 0.95. On the other hand,the 95% confidence interval for the odds ratio for the group who have a current status of moderate drinking does include 1, so the p-value for that comparison is greater than 0.05.
    Incorrect
    Since the 95% confidence interval for the odds ratio for the group with a lifetime history of moderate drinking does not include 1, the p-value for that comparison is less than 0.95. On the other hand,the 95% confidence interval for the odds ratio for the group who have a current status of moderate drinking does include 1, so the p-value for that comparison is greater than 0.05.
    We have to be careful how we interpret a p-value greater than 0.05. Revisit the Hypothesis Testing module if you need to recap this.
    Correct!
    Since the 95% confidence interval for the odds ratio for the group with a lifetime history of moderate drinking does not include 1, the p-value for that comparison is less than 0.95. On the other hand,the 95% confidence interval for the odds ratio for the group who have a current status of moderate drinking does include 1, so the p-value for that comparison is greater than 0.05.
    Consequently, we can conclude that a lifetime history of moderate drinking reduces amyloid-beta deposits in the brain. On the other hand, we can draw no definitive conclusions concerning the effects of a current status of moderate drinking, since the p-value is greater than 0.05.

  2. A vaccine development lab is testing a new vaccine for COVID-19. They recruit two cohorts, one injected with the vaccine and one with a placebo, and follow the subjects for a period of 6 months. They compute the odds ratio for COVID-19 infection and report it along with a 95% confidence interval.

    If instead they had computed and reported a 99% confidence interval, would this interval be wider (contain more values) or narrower (contain fewer values) than the 95% confidence interval?

    1. If the odds ratio were less than 1, the 99% confidence interval would be narrower, but if the odds ratio were more than 1, it would be wider.
    2. If the odds ratio were less than 1, the 99% confidence interval would be wider, but if the odds ratio were more than 1, it would be narrower.
    3. The 99% confidence interval would be narrower than the 95% confidence interval.
    4. The 99% confidence interval would be wider than the 95% confidence interval.
    5. The 99% confidence interval could be wider or narrower than the 95% confidence interval, depending on the data.
    Incorrect
    Increasing our level of confidence requires choosing an interval we are more certain captures the true odds ratio. To do this, we must include more values in the interval, so a 99% confidence interval would be wider than a 95% confidence interval. This is true regardless of the estimated odds ratio, or any other facets of the data.
    Correct!
    Increasing our level of confidence requires choosing an interval we are more certain captures the true odds ratio. To do this, we must include more values in the interval, so a 99% confidence interval would be wider than a 95% confidence interval.