  One of the long-standing challenges in lexical semantics consists in learning
representations of words which reflect their semantic properties. The
remarkable success of word embeddings for this purpose suggests that
high-quality representations can be obtained by summarizing the sentence
contexts of word mentions. In this paper, we propose a method for learning word
representations that follows this basic strategy, but differs from standard
word embeddings in two important ways. First, we take advantage of
contextualized language models (CLMs) rather than bags of word vectors to
encode contexts. Second, rather than learning a word vector directly, we use a
topic model to partition the contexts in which words appear, and then learn
different topic-specific vectors for each word. Finally, we use a task-specific
supervision signal to make a soft selection of the resulting vectors. We show
that this simple strategy leads to high-quality word vectors, which are more
predictive of semantic properties than word embeddings and existing CLM-based
strategies.
