  One of the major drawbacks of deep learning models for computer vision has
been their inability to retain multiple sources of information in a modular
fashion. For instance, given a network that has been trained on a source task,
we would like to re-train this network on a similar, yet different, target task
while maintaining its performance on the source task. Simultaneously,
researchers have extensively studied modularization of network weights to
localize and identify the set of weights culpable for eliciting the observed
performance on a given task. One set of works studies the modularization
induced in the weights of a neural network by learning and analysing weight
masks. In this work, we combine these fields to study three such weight masking
methods and analyse their ability to mitigate "forgetting'' on the source task
while also allowing for efficient finetuning on the target task. We find that
different masking techniques have trade-offs in retaining knowledge in the
source task without adversely affecting target task performance.
