  Observational studies are needed when experiments are not possible. Within
study comparisons (WSC) compare observational and experimental estimates that
test the same hypothesis using the same treatment group, outcome, and estimand.
Meta-analyzing 39 of them, we compare mean bias and its variance for the eight
observational designs that result from combining whether there is a pretest
measure of the outcome or not, whether the comparison group is local to the
treatment group or not, and whether there is a relatively rich set of other
covariates or not. Of these eight designs, one combines all three design
elements, another has none, and the remainder include any one or two. We found
that both the mean and variance of bias decline as design elements are added,
with the lowest mean and smallest variance in a design with all three elements.
The probability of bias falling within 0.10 standard deviations of the
experimental estimate varied from 59 to 83 percent in Bayesian analyses and
from 86 to 100 percent in non-Bayesian ones -- the ranges depending on the
level of data aggregation. But confounding remains possible due to each of the
eight observational study design cells including a different set of WSC
studies.
