  In this paper, we consider the problem of improving the inference latency of
language model-based dense retrieval systems by introducing structural
compression and model size asymmetry between the context and query encoders.
First, we investigate the impact of pre and post-training compression on the
MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that
asymmetry in the dual encoders in dense retrieval can lead to improved
inference efficiency. Knowing this, we introduce Kullback Leibler Alignment of
Embeddings (KALE), an efficient and accurate method for increasing the
inference efficiency of dense retrieval methods by pruning and aligning the
query encoder after training. Specifically, KALE extends traditional Knowledge
Distillation after bi-encoder training, allowing for effective query encoder
compression without full retraining or index generation. Using KALE and
asymmetric training, we can generate models which exceed the performance of
DistilBERT despite having 3x faster inference.
