  The Word Embedding Association Test shows that GloVe and word2vec word
embeddings exhibit human-like implicit biases based on gender, race, and other
social constructs (Caliskan et al., 2017). Meanwhile, research on learning
reusable text representations has begun to explore sentence-level texts, with
some sentence encoders seeing enthusiastic adoption. Accordingly, we extend the
Word Embedding Association Test to measure bias in sentence encoders. We then
test several sentence encoders, including state-of-the-art methods such as ELMo
and BERT, for the social biases studied in prior work and two important biases
that are difficult or impossible to test at the word level. We observe mixed
results including suspicious patterns of sensitivity that suggest the test's
assumptions may not hold in general. We conclude by proposing directions for
future work on measuring bias in sentence encoders.
