  In this work, we demonstrate that the contextualized word vectors derived
from pretrained masked language model-based encoders share a common, perhaps
undesirable pattern across layers. Namely, we find cases of persistent outlier
neurons within BERT and RoBERTa's hidden state vectors that consistently bear
the smallest or largest values in said vectors. In an attempt to investigate
the source of this information, we introduce a neuron-level analysis method,
which reveals that the outliers are closely related to information captured by
positional embeddings. We also pre-train the RoBERTa-base models from scratch
and find that the outliers disappear without using positional embeddings. These
outliers, we find, are the major cause of anisotropy of encoders' raw vector
spaces, and clipping them leads to increased similarity across vectors. We
demonstrate this in practice by showing that clipped vectors can more
accurately distinguish word senses, as well as lead to better sentence
embeddings when mean pooling. In three supervised tasks, we find that clipping
does not affect the performance.
