  Although models using contextual word embeddings have achieved
state-of-the-art results on a host of NLP tasks, little is known about exactly
what information these embeddings encode about the context words that they are
understood to reflect. To address this question, we introduce a suite of
probing tasks that enable fine-grained testing of contextual embeddings for
encoding of information about surrounding words. We apply these tasks to
examine the popular BERT, ELMo and GPT contextual encoders, and find that each
of our tested information types is indeed encoded as contextual information
across tokens, often with near-perfect recoverability-but the encoders vary in
which features they distribute to which tokens, how nuanced their distributions
are, and how robust the encoding of each feature is to distance. We discuss
implications of these results for how different types of models breakdown and
prioritize word-level context information when constructing token embeddings.
