  Life-long learning aims at learning a sequence of tasks without forgetting
the previously acquired knowledge. However, the involved training data may not
be life-long legitimate due to privacy or copyright reasons. In practical
scenarios, for instance, the model owner may wish to enable or disable the
knowledge of specific tasks or specific samples from time to time. Such
flexible control over knowledge transfer, unfortunately, has been largely
overlooked in previous incremental or decremental learning methods, even at a
problem-setup level. In this paper, we explore a novel learning scheme, termed
as Learning wIth Recoverable Forgetting (LIRF), that explicitly handles the
task- or sample-specific knowledge removal and recovery. Specifically, LIRF
brings in two innovative schemes, namely knowledge deposit and withdrawal,
which allow for isolating user-designated knowledge from a pre-trained network
and injecting it back when necessary. During the knowledge deposit process, the
specified knowledge is extracted from the target network and stored in a
deposit module, while the insensitive or general knowledge of the target
network is preserved and further augmented. During knowledge withdrawal, the
taken-off knowledge is added back to the target network. The deposit and
withdraw processes only demand for a few epochs of finetuning on the removal
data, ensuring both data and time efficiency. We conduct experiments on several
datasets, and demonstrate that the proposed LIRF strategy yields encouraging
results with gratifying generalization capability.
