  As deep learning models are becoming larger and data-hungrier, there are
growing ethical, legal and technical concerns over use of data: in practice,
agreements on data use may change over time, rendering previously-used training
data impermissible for training purposes. These issues have driven increased
attention to machine unlearning: removing "the influence of" a subset of
training data from a trained model. In this work, we advocate for a relaxed
definition of unlearning that does not address privacy applications but targets
a scenario where a data owner withdraws permission of use of their data for
training purposes. In this context, we consider the important problem of
\emph{transfer unlearning} where a pretrained model is transferred to a target
dataset that contains some "non-static" data that may need to be unlearned in
the future. We propose a new method that uses a mechanism for selecting
relevant examples from an auxiliary "static" dataset, and finetunes on the
selected data instead of "non-static" target data; addressing all unlearning
requests ahead of time. We also adapt a recent relaxed definition of unlearning
to our problem setting and demonstrate that our approach is an exact transfer
unlearner according to it, while being highly efficient (amortized). We find
that our method outperforms the gold standard "exact unlearning" (finetuning on
only the "static" portion of the target dataset) on several datasets,
especially for small "static" sets, sometimes approaching an upper bound for
test accuracy. We also analyze factors influencing the accuracy boost obtained
by data selection.
