  Observational studies are valuable for estimating the effects of various
medical interventions, but are notoriously difficult to evaluate because the
methods used in observational studies require many untestable assumptions. This
lack of verifiability makes it difficult both to compare different
observational study methods and to trust the results of any particular
observational study. In this work, we propose TrialVerify, a new approach for
evaluating observational study methods based on ground truth sourced from
clinical trial reports. We process trial reports into a denoised collection of
known causal relationships that can then be used to estimate the precision and
recall of various observational study methods. We then use TrialVerify to
evaluate multiple observational study methods in terms of their ability to
identify the known causal relationships from a large national insurance claims
dataset. We found that inverse propensity score weighting is an effective
approach for accurately reproducing known causal relationships and outperforms
other observational study methods. TrialVerify is made freely available for
others to evaluate observational study methods.
