  Pretrained language model (PLM) hidden states are frequently employed as
contextual word embeddings (CWE): high-dimensional representations that encode
semantic information given linguistic context. Across many areas of
computational linguistics research, similarity between CWEs is interpreted as
semantic similarity. However, it remains unclear exactly what information is
encoded in PLM hidden states. We investigate this practice by probing PLM
representations using minimal orthographic noise. We expect that if CWEs
primarily encode semantic information, a single character swap in the input
word will not drastically affect the resulting representation,given sufficient
linguistic context. Surprisingly, we find that CWEs generated by popular PLMs
are highly sensitive to noise in input data, and that this sensitivity is
related to subword tokenization: the fewer tokens used to represent a word at
input, the more sensitive its corresponding CWE. This suggests that CWEs
capture information unrelated to word-level meaning and can be manipulated
through trivial modifications of input data. We conclude that these PLM-derived
CWEs may not be reliable semantic proxies, and that caution is warranted when
interpreting representational similarity
