A clinical trial proves a drug works by beating a precise statement that it does nothing. Researchers write down that “no effect” claim, called the null hypothesis, before the trial starts, and the statistics test whether the results let them reject it. The drug nintedanib, appraised by England’s National Institute for Health and Care Excellence (NICE) in 2016, shows how that plays out.
On its main measure the drug cleared the bar easily. On two others that patients would care about, it did not.
Setting up the test
Every trial is built around a hypothesis, and that hypothesis sets the trial’s primary objective, write Stephanie Pugh and colleagues in the Journal of Nuclear Medicine. The null hypothesis is a narrow statement about the population, such as two treatments performing the same. The alternative hypothesis is the broader claim the researchers hope to show.
The trial itself only studies a sample, a subset of the people the drug is meant for. Statistics estimate the true figure for that whole population from the sample. BBC Bitesize, a schools revision guide, notes that a small group raises the chance that some unknown factor skews the result.
The two ways to be wrong
Two errors frame every result. A type I error is rejecting the null hypothesis when it is in fact true: declaring a drug works when it does not. A type II error is failing to reject it when it is false, and so missing a drug that does work.
What nintedanib showed
Three trials tested nintedanib against a placebo for 52 weeks in adults aged 40 or over with idiopathic pulmonary fibrosis. A third, smaller dose-finding trial called TOMORROW enrolled 428. The two phase 3 trials were INPULSIS 1, with 513 people, and INPULSIS 2, with 548. All three were randomised and double-blind, so neither patients nor doctors knew who got the drug. Each also set entry rules: patients needed a lung capacity of at least 50 per cent of the normal value for their profile.
The primary outcome was how fast patients lost FVC, a standard lung test of the volume of air they can blow out, measured in millilitres a year. On nintedanib the decline was 114.7 ml a year, about half the placebo rate. The difference was statistically significant, at p<0.001.
Where it fell short
Fewer patients died on the drug than on placebo, but the difference was not statistically significant.
Flare-ups, called acute exacerbations, gave a split verdict. INPULSIS 2 found a significant benefit and INPULSIS 1 found none. Pooled, 4.9 per cent of patients on nintedanib had at least one in 52 weeks against 7.6 per cent on placebo, with a p-value of 0.08, which NICE recorded as not significant.
The maker, Boehringer Ingelheim, noted that the trials were not powered to detect an effect on exacerbations. In other words, they enrolled too few patients to stand a fair chance of spotting one, which is the territory of the type II error.
Why testing more outcomes is a trap
Measuring many things raises a hazard. Each extra test adds another chance of a false positive, and Pugh and colleagues say a trial testing several hypotheses must correct for that inflation of type I error.
The method also decides between a one-sided test, which checks for a difference in one direction only, and a two-sided test, which checks both.
Nintedanib’s lung-function result passed, and with room to spare. Its survival and flare-up figures pointed the right way but did not clear the bar, and NICE’s write-up keeps the two kinds of result apart.
