  Transfer learning is widely used for training deep neural networks (DNN) for
building a powerful representation. Even after the pre-trained model is adapted
for the target task, the representation performance of the feature extractor is
retained to some extent. As the performance of the pre-trained model can be
considered the private property of the owner, it is natural to seek the
exclusive right of the generalized performance of the pre-trained weight. To
address this issue, we suggest a new paradigm of transfer learning called
disposable transfer learning (DTL), which disposes of only the source task
without degrading the performance of the target task. To achieve knowledge
disposal, we propose a novel loss named Gradient Collision loss (GC loss). GC
loss selectively unlearns the source knowledge by leading the gradient vectors
of mini-batches in different directions. Whether the model successfully
unlearns the source task is measured by piggyback learning accuracy (PL
accuracy). PL accuracy estimates the vulnerability of knowledge leakage by
retraining the scrubbed model on a subset of source data or new downstream
data. We demonstrate that GC loss is an effective approach to the DTL problem
by showing that the model trained with GC loss retains the performance on the
target task with a significantly reduced PL accuracy.
