  Online reinforcement learning and other adaptive sampling algorithms are
increasingly used in digital intervention experiments to optimize treatment
delivery for users over time. In this work, we focus on longitudinal user data
collected by a large class of adaptive sampling algorithms that are designed to
optimize treatment decisions online using accruing data from multiple users.
Combining or "pooling" data across users allows adaptive sampling algorithms to
potentially learn faster. However, by pooling, these algorithms induce
dependence between the sampled user data trajectories; we show that this can
cause standard variance estimators for i.i.d. data to underestimate the true
variance of common estimators on this data type. We develop novel methods to
perform a variety of statistical analyses on such adaptively sampled data via
Z-estimation. Specifically, we introduce the \textit{adaptive} sandwich
variance estimator, a corrected sandwich estimator that leads to consistent
variance estimates under adaptive sampling. Additionally, to prove our results
we develop novel theoretical tools for empirical processes on non-i.i.d.,
adaptively sampled longitudinal data which may be of independent interest. This
work is motivated by our efforts in designing experiments in which online
reinforcement learning algorithms optimize treatment decisions, yet statistical
inference is essential for conducting analyses after experiments conclude.
