  Large pretrained language models (PreLMs) are revolutionizing natural
language processing across all benchmarks. However, their sheer size is
prohibitive for small laboratories or for deployment on mobile devices.
Approaches like pruning and distillation reduce the model size but typically
retain the same model architecture. In contrast, we explore distilling PreLMs
into a different, more efficient architecture, Continual Multiplication of
Words (CMOW), which embeds each word as a matrix and uses matrix multiplication
to encode sequences. We extend the CMOW architecture and its CMOW/CBOW-Hybrid
variant with a bidirectional component for more expressive power, per-token
representations for a general (task-agnostic) distillation during pretraining,
and a two-sequence encoding scheme that facilitates downstream tasks on
sentence pairs, such as sentence similarity and natural language inference. Our
matrix-based bidirectional CMOW/CBOW-Hybrid model is competitive to DistilBERT
on question similarity and recognizing textual entailment, but uses only half
of the number of parameters and is three times faster in terms of inference
speed. We match or exceed the scores of ELMo for all tasks of the GLUE
benchmark except for the sentiment analysis task SST-2 and the linguistic
acceptability task CoLA. However, compared to previous cross-architecture
distillation approaches, we demonstrate a doubling of the scores on detecting
linguistic acceptability. This shows that matrix-based embeddings can be used
to distill large PreLM into competitive models and motivates further research
in this direction.
