  The increasing deployment of artificial intelligence (AI) tools to inform
decision making across diverse areas including healthcare, employment, social
benefits, and government policy, presents a serious risk for disabled people,
who have been shown to face bias in AI implementations. While there has been
significant work on analysing and mitigating algorithmic bias, the broader
mechanisms of how bias emerges in AI applications are not well understood,
hampering efforts to address bias where it begins. In this article, we
illustrate how bias in AI-assisted decision making can arise from a range of
specific design decisions, each of which may seem self-contained and
non-biasing when considered separately. These design decisions include basic
problem formulation, the data chosen for analysis, the use the AI technology is
put to, and operational design elements in addition to the core algorithmic
design. We draw on three historical models of disability common to different
decision-making settings to demonstrate how differences in the definition of
disability can lead to highly distinct decisions on each of these aspects of
design, leading in turn to AI technologies with a variety of biases and
downstream effects. We further show that the potential harms arising from
inappropriate definitions of disability in fundamental design stages are
further amplified by a lack of transparency and disabled participation
throughout the AI design process. Our analysis provides a framework for
critically examining AI technologies in decision-making contexts and guiding
the development of a design praxis for disability-related AI analytics. We put
forth this article to provide key questions to facilitate disability-led design
and participatory development to produce more fair and equitable AI
technologies in disability-related contexts.
