  Extracting dense representations for terms and phrases is a task of great
importance for knowledge discovery platforms targeting highly-technical fields.
Dense representations are used as features for downstream components and have
multiple applications ranging from ranking results in search to summarization.
Common approaches to create dense representations include training
domain-specific embeddings with self-supervised setups or using sentence
encoder models trained over similarity tasks. In contrast to static embeddings,
sentence encoders do not suffer from the out-of-vocabulary (OOV) problem, but
impose significant computational costs. In this paper, we propose a fully
unsupervised approach to text encoding that consists of training small
character-based models with the objective of reconstructing large pre-trained
embedding matrices. Models trained with this approach can not only match the
quality of sentence encoders in technical domains, but are 5 times smaller and
up to 10 times faster, even on high-end GPUs.
