  Adaptive collection of data is commonplace in applications throughout science
and engineering. From the point of view of statistical inference however,
adaptive data collection induces memory and correlation in the samples, and
poses significant challenge. We consider the high-dimensional linear
regression, where the samples are collected adaptively, and the sample size $n$
can be smaller than $p$, the number of covariates. In this setting, there are
two distinct sources of bias: the first due to regularization imposed for
consistent estimation, e.g. using the LASSO, and the second due to adaptivity
in collecting the samples. We propose "online debiasing", a general procedure
for estimators such as the LASSO, which addresses both sources of bias. In two
concrete contexts $(i)$ time series analysis and $(ii)$ batched data
collection, we demonstrate that online debiasing optimally debiases the LASSO
estimate when the underlying parameter $\theta_0$ has sparsity of order
$o(\sqrt{n}/\log p)$. In this regime, the debiased estimator can be used to
compute $p$-values and confidence intervals of optimal size.
