  Caseworkers in the child welfare (CW) sector use predictive decision-making
algorithms built on risk assessment (RA) data to guide and support CW
decisions. Researchers have highlighted that RAs can contain biased signals
which flatten CW case complexities and that the algorithms may benefit from
incorporating contextually rich case narratives, i.e. - casenotes written by
caseworkers. To investigate this hypothesized improvement, we quantitatively
deconstructed two commonly used RAs from a United States CW agency. We trained
classifier models to compare the predictive validity of RAs with and without
casenote narratives and applied computational text analysis on casenotes to
highlight topics uncovered in the casenotes. Our study finds that common risk
metrics used to assess families and build CWS predictive risk models (PRMs) are
unable to predict discharge outcomes for children who are not reunified with
their birth parent(s). We also find that although casenotes cannot predict
discharge outcomes, they contain contextual case signals. Given the lack of
predictive validity of RA scores and casenotes, we propose moving beyond
quantitative risk assessments for public sector algorithms and towards using
contextual sources of information such as narratives to study public
sociotechnical systems.
