  The carbon footprint of natural language processing research has been
increasing in recent years due to its reliance on large and inefficient neural
network implementations. Distillation is a network compression technique which
attempts to impart knowledge from a large model to a smaller one. We use
teacher-student distillation to improve the efficiency of the Biaffine
dependency parser which obtains state-of-the-art performance with respect to
accuracy and parsing speed (Dozat and Manning, 2017). When distilling to 20\%
of the original model's trainable parameters, we only observe an average
decrease of $\sim$1 point for both UAS and LAS across a number of diverse
Universal Dependency treebanks while being 2.30x (1.19x) faster than the
baseline model on CPU (GPU) at inference time. We also observe a small increase
in performance when compressing to 80\% for some treebanks. Finally, through
distillation we attain a parser which is not only faster but also more accurate
than the fastest modern parser on the Penn Treebank.
