  Estimators computed from adaptively collected data do not behave like their
non-adaptive brethren. Rather, the sequential dependence of the collection
policy can lead to severe distributional biases that persist even in the
infinite data limit. We develop a general method -- $\mathbf{W}$-decorrelation
-- for transforming the bias of adaptive linear regression estimators into
variance. The method uses only coarse-grained information about the data
collection policy and does not need access to propensity scores or exact
knowledge of the policy. We bound the finite-sample bias and variance of the
$\mathbf{W}$-estimator and develop asymptotically correct confidence intervals
based on a novel martingale central limit theorem. We then demonstrate the
empirical benefits of the generic $\mathbf{W}$-decorrelation procedure in two
different adaptive data settings: the multi-armed bandit and the autoregressive
time series.
