  This paper presents Domain-Specific Sub-network (DoSS). It uses a set of
masks obtained through pruning to define a sub-network for each domain and
finetunes the sub-network parameters on domain data. This performs very closely
and drastically reduces the number of parameters compared to finetuning the
whole network on each domain. Also a method to make masks unique per domain is
proposed and shown to greatly improve the generalization to unseen domains. In
our experiments on German to English machine translation the proposed method
outperforms the strong baseline of continue training on multi-domain (medical,
tech and religion) data by 1.47 BLEU points. Also continue training DoSS on new
domain (legal) outperforms the multi-domain (medical, tech, religion, legal)
baseline by 1.52 BLEU points.
