  Contextualized word embedding models, such as ELMo, generate meaningful
representations of words and their context. These models have been shown to
have a great impact on downstream applications. However, in many cases, the
contextualized embedding of a word changes drastically when the context is
paraphrased. As a result, the downstream model is not robust to paraphrasing
and other linguistic variations. To enhance the stability of contextualized
word embedding models, we propose an approach to retrofitting contextualized
embedding models with paraphrase contexts. Our method learns an orthogonal
transformation on the input space, which seeks to minimize the variance of word
representations on paraphrased contexts. Experiments show that the retrofitted
model significantly outperforms the original ELMo on various sentence
classification and language inference tasks.
