  Neural machine translation often adopts the fine-tuning approach to adapt to
specific domains. However, nonrestricted fine-tuning can easily degrade on the
general domain and over-fit to the target domain. To mitigate the issue, we
propose Prune-Tune, a novel domain adaptation method via gradual pruning. It
learns tiny domain-specific sub-networks during fine-tuning on new domains.
Prune-Tune alleviates the over-fitting and the degradation problem without
model modification. Furthermore, Prune-Tune is able to sequentially learn a
single network with multiple disjoint domain-specific sub-networks for multiple
domains. Empirical experiment results show that Prune-Tune outperforms several
strong competitors in the target domain test set without sacrificing the
quality on the general domain in both single and multi-domain settings. The
source code and data are available at https://github.com/ohlionel/Prune-Tune.
