  Algorithms are increasingly used to aid, or in some cases supplant, human
decision-making, particularly for decisions that hinge on predictions. As a
result, two additional features in addition to prediction quality have
generated interest: (i) to facilitate human interaction and understanding with
these algorithms, we desire prediction functions that are in some fashion
simple or interpretable; and (ii) because they influence consequential
decisions, we also want them to produce equitable allocations. We develop a
formal model to explore the relationship between the demands of simplicity and
equity. Although the two concepts appear to be motivated by qualitatively
distinct goals, we show a fundamental inconsistency between them. Specifically,
we formalize a general framework for producing simple prediction functions, and
in this framework we establish two basic results. First, every simple
prediction function is strictly improvable: there exists a more complex
prediction function that is both strictly more efficient and also strictly more
equitable. Put another way, using a simple prediction function both reduces
utility for disadvantaged groups and reduces overall welfare relative to other
options. Second, we show that simple prediction functions necessarily create
incentives to use information about individuals' membership in a disadvantaged
group --- incentives that weren't present before simplification, and that work
against these individuals. Thus, simplicity transforms disadvantage into bias
against the disadvantaged group. Our results are not only about algorithms but
about any process that produces simple models, and as such they connect to the
psychology of stereotypes and to an earlier economics literature on statistical
discrimination.
