  Claims from observational studies often fail to replicate. A study was
undertaken to assess the reliability of cohort studies used in a highly cited
meta-analysis of the association between ambient nitrogen dioxide, NO2, and
fine particulate matter, PM2.5, concentrations early in life and development of
asthma later in life. The numbers of statistical tests possible were estimated
for 19 base papers considered for the meta-analysis. A p-value plot for NO2 and
PM2.5 was constructed to evaluate effect heterogeneity of p-values used from
the base papers. The numbers of statistical tests possible in the base papers
were large - median 13,824, interquartile range 1,536-221,184; range 96-42M, in
comparison to statistical test results presented. Statistical test results
drawn from the base papers are unlikely to provide unbiased measures for
meta-analysis. The p-value plot indicated that heterogeneity of the NO2 results
across the base papers is consistent with a two-component mixture. First, it
makes no sense to average across a mixture in meta-analysis. Second, the shape
of the p-value plot for NO2 appears consistent with the possibility of analysis
manipulation to obtain small p-values in several of the cohort studies. As for
PM2.5, all corresponding p-values fall on a 45-degree line indicating complete
randomness rather than a true association. Our interpretation of the
meta-analysis is that the random p-values indicating no cause-effect
associations are more plausible and that their meta-analysis will not likely
replicate in the absence of bias. We conclude that claims made in the base
papers used for meta-analysis are unreliable due to bias induced by multiple
testing and multiple modelling, MTMM. We also show there is evidence that the
heterogeneity across the base papers used for meta-analysis is more complex
than simple sampling from a normal process.
