  Deep Neural Networks (DNN) could forget the knowledge about earlier tasks
when learning new tasks, and this is known as \textit{catastrophic forgetting}.
While recent continual learning methods are capable of alleviating the
catastrophic problem on toy-sized datasets, some issues still remain to be
tackled when applying them in real-world problems. Recently, the fast
mask-based learning method (e.g. piggyback \cite{mallya2018piggyback}) is
proposed to address these issues by learning only a binary element-wise mask in
a fast manner, while keeping the backbone model fixed. However, the binary mask
has limited modeling capacity for new tasks. A more recent work
\cite{hung2019compacting} proposes a compress-grow-based method (CPG) to
achieve better accuracy for new tasks by partially training backbone model, but
with order-higher training cost, which makes it infeasible to be deployed into
popular state-of-the-art edge-/mobile-learning. The primary goal of this work
is to simultaneously achieve fast and high-accuracy multi task adaption in
continual learning setting. Thus motivated, we propose a new training method
called \textit{kernel-wise Soft Mask} (KSM), which learns a kernel-wise hybrid
binary and real-value soft mask for each task, while using the same backbone
model. Such a soft mask can be viewed as a superposition of a binary mask and a
properly scaled real-value tensor, which offers a richer representation
capability without low-level kernel support to meet the objective of low
hardware overhead. We validate KSM on multiple benchmark datasets against
recent state-of-the-art methods (e.g. Piggyback, Packnet, CPG, etc.), which
shows good improvement in both accuracy and training cost.
