Combining the results from several experiments

Suppose that we have carried out an experiment to compare a standard/reference diet and 5 other diets (Standard, D2, D3, D4, D5 and D6). We find that D2 is better than the Standard diet, but given the variability and the numbers of experimental units (animals) on each diet the difference was not statistically significant at the 5% level, which we had previously decided to be our cut-off point for considering an effect to be real (statistically significant) or not. But nonetheless there was a 'tendancy'. Suppose that we had obtained:

D2 - Standard = 5.20 ± 3 Pr = ~.10 => -2 ln Pr = 4.6052

In a previous experimental trial the previous year another budding graduate student had used the same Standard diet and D2, as well as other diets, and obtained:

D2 - Standard = 5.00 ± 2.5 Pr = ~.055 => -2 ln Pr = 5.8008

In reviewing the literature we find that another researcher at another University has published a paper and amongst other comparisons has also used the Standard diet and D2, and obtained:

D2 - Standard = 4.5 ± 3 Pr = ~.18 => -2 ln Pr = 3.4296

None of these results are statistically significant, but they all tend in the same direction; suggesting that Diet 2 is superior to the Standard. Can we combine these results?

Yes, Fisher (1950) showed that -2 ln P   is distributed as a c with 2 degrees of freedom. So, if the trials are indeed seperate, completely independent, we can add these c together to produce a pooled combined result.

See Steel, Torrie and Dickey, Chapter 20.5.

-2 ln P (c2)
So 4.6052

The critical, tabulated value for a c2 with 6 degrees of freedom at the 5% level is 12.6. Why 6 d.f.? Because we are combining 3 trials, each with 2 degrees of freedom for the c2. It is irrelevant how many experimental units (animals, or plants, or whatever) and degrees of freedom there were for each of the trials. We are not combining the parameter estimates, we are combining the probabilities.

From the above we would reject the Null Hypothesis, that the 2 diets are equal and we would conclude that they differ; that D2 is indeed better than the Standard diet.

Note the additional information that we have been able to glean, or extract, from 3 non-significant results.

R.I. Cue ©
Department of Animal Science, McGill University
last update : 2010 May 4