  Recent research increasingly brings to question the appropriateness of using
predictive tools in complex, real-world tasks. While a growing body of work has
explored ways to improve value alignment in these tools, comparatively less
work has centered concerns around the fundamental justifiability of using these
tools. This work seeks to center validity considerations in deliberations
around whether and how to build data-driven algorithms in high-stakes domains.
Toward this end, we translate key concepts from validity theory to predictive
algorithms. We apply the lens of validity to re-examine common challenges in
problem formulation and data issues that jeopardize the justifiability of using
predictive algorithms and connect these challenges to the social science
discourse around validity. Our interdisciplinary exposition clarifies how these
concepts apply to algorithmic decision making contexts. We demonstrate how
these validity considerations could distill into a series of high-level
questions intended to promote and document reflections on the legitimacy of the
predictive task and the suitability of the data.
