  Artificial intelligence (AI) hiring tools have revolutionized resume
screening, and large language models (LLMs) have the potential to do the same.
However, given the biases which are embedded within LLMs, it is unclear whether
they can be used in this scenario without disadvantaging groups based on their
protected attributes. In this work, we investigate the possibilities of using
LLMs in a resume screening setting via a document retrieval framework that
simulates job candidate selection. Using that framework, we then perform a
resume audit study to determine whether a selection of Massive Text Embedding
(MTE) models are biased in resume screening scenarios. We simulate this for
nine occupations, using a collection of over 500 publicly available resumes and
500 job descriptions. We find that the MTEs are biased, significantly favoring
White-associated names in 85.1\% of cases and female-associated names in only
11.1\% of cases, with a minority of cases showing no statistically significant
differences. Further analyses show that Black males are disadvantaged in up to
100\% of cases, replicating real-world patterns of bias in employment settings,
and validate three hypotheses of intersectionality. We also find an impact of
document length as well as the corpus frequency of names in the selection of
resumes. These findings have implications for widely used AI tools that are
automating employment, fairness, and tech policy.
