  Motivated by the EU's "Right To Be Forgotten" regulation, we initiate a study
of statistical data deletion problems where users' data are accessible only for
a limited period of time. This setting is formulated as an online supervised
learning task with \textit{constant memory limit}. We propose a deletion-aware
algorithm \texttt{FIFD-OLS} for the low dimensional case, and witness a
catastrophic rank swinging phenomenon due to the data deletion operation, which
leads to statistical inefficiency. As a remedy, we propose the
\texttt{FIFD-Adaptive Ridge} algorithm with a novel online regularization
scheme, that effectively offsets the uncertainty from deletion. In theory, we
provide the cumulative regret upper bound for both online forgetting
algorithms. In the experiment, we showed \texttt{FIFD-Adaptive Ridge}
outperforms the ridge regression algorithm with fixed regularization level, and
hopefully sheds some light on more complex statistical models.
