  Although much literature has established the presence of demographic bias in
natural language processing (NLP) models, most work relies on curated bias
metrics that may not be reflective of real-world applications. At the same
time, practitioners are increasingly using algorithmic tools in high-stakes
settings, with particular recent interest in NLP. In this work, we focus on one
such setting: child protective services (CPS). CPS workers often write copious
free-form text notes about families they are working with, and CPS agencies are
actively seeking to deploy NLP models to leverage these data. Given
well-established racial bias in this setting, we investigate possible ways
deployed NLP is liable to increase racial disparities. We specifically examine
word statistics within notes and algorithmic fairness in risk prediction,
coreference resolution, and named entity recognition (NER). We document
consistent algorithmic unfairness in NER models, possible algorithmic
unfairness in coreference resolution models, and little evidence of exacerbated
racial bias in risk prediction. While there is existing pronounced criticism of
risk prediction, our results expose previously undocumented risks of racial
bias in realistic information extraction systems, highlighting potential
concerns in deploying them, even though they may appear more benign. Our work
serves as a rare realistic examination of NLP algorithmic fairness in a
potential deployed setting and a timely investigation of a specific risk
associated with deploying NLP in CPS settings.
