  There is a long-standing debate in the statistical, epidemiological and
econometric fields as to whether nonparametric estimation that uses
data-adaptive methods, like machine learning algorithms in model fitting,
confer any meaningful advantage over simpler, parametric approaches in
real-world, finite sample estimation of causal effects. We address the
question: when trying to estimate the effect of a treatment on an outcome,
across a universe of reasonable data distributions, how much does the choice of
nonparametric vs.~parametric estimation matter? Instead of answering this
question with simulations that reflect a few chosen data scenarios, we propose
a novel approach evaluating performance across thousands of data-generating
mechanisms drawn from non-parametric models with semi-informative priors. We
call this approach a Universal Monte-Carlo Simulation. We compare performance
of estimating the average treatment effect across two parametric estimators (a
g-computation estimator that uses a parametric outcome model and an inverse
probability of treatment weighted estimator) and two nonparametric estimators
(Bayesian additive regression trees and a targeted minimum loss-based estimator
that uses an ensemble of machine learning algorithms in model fitting). We
summarize estimator performance in terms of bias, confidence interval coverage,
and mean squared error. We find that the nonparametric estimators nearly always
outperform the parametric estimators with the exception of having similar
performance in terms of bias and similar-to-slightly-worse performance in terms
of coverage under the smallest sample size of N=100.
