  Much evidence in comparative effectiveness research is based on observational
studies. Researchers who conduct observational studies typically assume that
there are no unobservable differences between the treated and control groups.
Treatment effects are estimated after adjusting for observed differences
between treated and controls. However, treatment effect estimates may be biased
due to model misspecification. That is, if the method of treatment effect
estimation imposes unduly strong functional form assumptions, treatment effect
estimates may be significantly biased. In this study, we compare the
performance of a wide variety of treatment effect estimation methods. We do so
within the context of the REFLUX study from the UK. In REFLUX, after study
qualification, participants were enrolled in either a randomized trial arm or
patient preference arm. In the randomized trial, patients were randomly
assigned to either surgery or medical management. In the patient preference
arm, participants selected to either have surgery or medical management. We
attempt to recover the treatment effect estimate from the randomized trial arm
using the data from the patient preference arm of the study. We vary the method
of treatment effect estimation and record which methods are successful and
which are not. We apply over 20 different methods including standard regression
models as well as advanced machine learning methods. We find that simple
propensity score matching methods perform the worst. We also find significant
variation in performance across methods. The wide variation in performance
suggests analysts should use multiple methods of estimation as a robustness
check.
