  State-level policy evaluations commonly employ a difference-in-differences
(DID) study design; yet within this framework, statistical model specification
varies notably across studies. Motivated by applied state-level opioid policy
evaluations, this simulation study compares statistical performance of multiple
variations of two-way fixed effect models traditionally used for DID under a
range of simulation conditions. While most linear models resulted in minimal
bias, non-linear models and population-weighted versions of classic linear
two-way fixed effect and linear GEE models yielded considerable bias (60 to
160%). Further, root mean square error is minimized by linear AR models when
examining crude mortality rates and by negative binomial models when examining
raw death counts. In the context of frequentist hypothesis testing, many models
yielded high Type I error rates and very low rates of correctly rejecting the
null hypothesis (< 10%), raising concerns of spurious conclusions about policy
effectiveness. When considering performance across models, the linear
autoregressive models were optimal in terms of directional bias, root mean
squared error, Type I error, and correct rejection rates. These findings
highlight notable limitations of traditional statistical models commonly used
for DID designs, designs widely used in opioid policy studies and in state
policy evaluations more broadly.
