  Recent legislation of the "right to be forgotten" has led to the interest in
machine unlearning, where the learned models are endowed with the function to
forget information about specific training instances as if they have never
existed in the training set. Previous work mainly focuses on computer vision
scenarios and largely ignores the essentials of unlearning in NLP field, where
text data contains more explicit and sensitive personal information than
images. In this paper, we propose a general unlearning framework called KGA to
induce forgetfulness. Different from previous work that tries to recover
gradients or forces models to perform close to one specific distribution, KGA
maintains distribution differences (i.e., knowledge gap). This relaxes the
distribution assumption. Furthermore, we first apply the unlearning method to
various NLP tasks (i.e., classification, translation, response generation) and
propose several unlearning evaluation metrics with pertinence. Experiments on
large-scale datasets show that KGA yields comprehensive improvements over
baselines, where extensive analyses further validate the effectiveness of KGA
and provide insight into unlearning for NLP tasks.
