  Word embeddings carry stereotypical connotations from the text they are
trained on, which can lead to invalid inferences in downstream models that rely
on them. We use this observation to design a mechanism for measuring
stereotypes using the task of natural language inference. We demonstrate a
reduction in invalid inferences via bias mitigation strategies on static word
embeddings (GloVe). Further, we show that for gender bias, these techniques
extend to contextualized embeddings when applied selectively only to the static
components of contextualized embeddings (ELMo, BERT).
