  This research introduces the Multilevel Embedding Association Test (ML-EAT),
a method designed for interpretable and transparent measurement of intrinsic
bias in language technologies. The ML-EAT addresses issues of ambiguity and
difficulty in interpreting the traditional EAT measurement by quantifying bias
at three levels of increasing granularity: the differential association between
two target concepts with two attribute concepts; the individual effect size of
each target concept with two attribute concepts; and the association between
each individual target concept and each individual attribute concept. Using the
ML-EAT, this research defines a taxonomy of EAT patterns describing the nine
possible outcomes of an embedding association test, each of which is associated
with a unique EAT-Map, a novel four-quadrant visualization for interpreting the
ML-EAT. Empirical analysis of static and diachronic word embeddings, GPT-2
language models, and a CLIP language-and-image model shows that EAT patterns
add otherwise unobservable information about the component biases that make up
an EAT; reveal the effects of prompting in zero-shot models; and can also
identify situations when cosine similarity is an ineffective metric, rendering
an EAT unreliable. Our work contributes a method for rendering bias more
observable and interpretable, improving the transparency of computational
investigations into human minds and societies.
