  It is generally acknowledged that claims from observational studies often
fail to replicate. An exploratory study was undertaken to assess the
reliability of base studies used in meta-analysis of short-term air
quality-myocardial infarction risk and to judge the reliability of statistical
evidence from meta-analysis that uses data from observational studies. A highly
cited meta-analysis paper examining whether short-term air quality exposure
triggers myocardial infarction was evaluated as a case study. The paper
considered six air quality components - carbon monoxide, nitrogen dioxide,
sulfur dioxide, particulate matter 10 and 2.5 micrometers in diameter (PM10 and
PM2.5), and ozone. The number of possible questions and statistical models at
issue in each of 34 base papers used were estimated and p-value plots for each
of the air components were constructed to evaluate the effect heterogeneity of
p-values used from the base papers. Analysis search spaces (number of
statistical tests possible) in the base papers were large, median of 12,288,
interquartile range: 2,496 to 58,368, in comparison to actual statistical test
results presented. Statistical test results taken from the base papers may not
provide unbiased measures of effect for meta-analysis. Shapes of p-value plots
for the six air components were consistent with the possibility of analysis
manipulation to obtain small p-values in several base papers. Results suggest
the appearance of heterogeneous, researcher-generated p-values used in the
meta-analysis rather than unbiased evidence of real effects for air quality. We
conclude that this meta-analysis does not provide reliable evidence for an
association of air quality components with myocardial risk.
