  A recent trend in Natural Language Processing is the exponential growth in
Language Model (LM) size, which prevents research groups without a necessary
hardware infrastructure from participating in the development process. This
study investigates methods for Knowledge Distillation (KD) to provide efficient
alternatives to large-scale models. In this context, KD means extracting
information about language encoded in a Neural Network and Lexical Knowledge
Databases. We developed two methods to test our hypothesis that efficient
architectures can gain knowledge from LMs and extract valuable information from
lexical sources. First, we present a technique to learn confident probability
distribution for Masked Language Modeling by prediction weighting of multiple
teacher networks. Second, we propose a method for Word Sense Disambiguation
(WSD) and lexical KD that is general enough to be adapted to many LMs. Our
results show that KD with multiple teachers leads to improved training
convergence. When using our lexical pre-training method, LM characteristics are
not lost, leading to increased performance in Natural Language Understanding
(NLU) tasks over the state-of-the-art while adding no parameters. Moreover, the
improved semantic understanding of our model increased the task performance
beyond WSD and NLU in a real-problem scenario (Plagiarism Detection). This
study suggests that sophisticated training methods and network architectures
can be superior over scaling trainable parameters. On this basis, we suggest
the research area should encourage the development and use of efficient models
and rate impacts resulting from growing LM size equally against task
performance.
