  Adapting pre-trained neural models to downstream tasks has become the
standard practice for obtaining high-quality models. In this work, we propose a
novel model adaptation paradigm, adapting by pruning, which prunes neural
connections in the pre-trained model to optimise the performance on the target
task; all remaining connections have their weights intact. We formulate
adapting-by-pruning as an optimisation problem with a differentiable loss and
propose an efficient algorithm to prune the model. We prove that the algorithm
is near-optimal under standard assumptions and apply the algorithm to adapt
BERT to some GLUE tasks. Results suggest that our method can prune up to 50%
weights in BERT while yielding similar performance compared to the fine-tuned
full model. We also compare our method with other state-of-the-art pruning
methods and study the topological differences of their obtained sub-networks.
