  Simulators make unique benchmarks for causal effect estimation since they do
not rely on unverifiable assumptions or the ability to intervene on real-world
systems, but are often too simple to capture important aspects of real
applications. We propose a simulator of Alzheimer's disease aimed at modeling
intricacies of healthcare data while enabling benchmarking of causal effect and
policy estimators. We fit the system to the Alzheimer's Disease Neuroimaging
Initiative (ADNI) dataset and ground hand-crafted components in results from
comparative treatment trials and observational treatment patterns. The
simulator includes parameters which alter the nature and difficulty of the
causal inference tasks, such as latent variables, effect heterogeneity, length
of observed history, behavior policy and sample size. We use the simulator to
compare estimators of average and conditional treatment effects.
