  Contextualized word embeddings in language models have given much advance to
NLP. Intuitively, sentential information is integrated into the representation
of words, which can help model polysemy. However, context sensitivity also
leads to the variance of representations, which may break the semantic
consistency for synonyms. We quantify how much the contextualized embeddings of
each word sense vary across contexts in typical pre-trained models. Results
show that contextualized embeddings can be highly consistent across contexts.
In addition, part-of-speech, number of word senses, and sentence length have an
influence on the variance of sense representations. Interestingly, we find that
word representations are position-biased, where the first words in different
contexts tend to be more similar. We analyze such a phenomenon and also propose
a simple way to alleviate such bias in distance-based word sense disambiguation
settings.
