  Managers, employers, policymakers, and others often seek to understand
whether decisions are biased against certain groups. One popular analytic
strategy is to estimate disparities after adjusting for observed covariates,
typically with a regression model. This approach, however, suffers from two key
statistical challenges. First, omitted-variable bias can skew results if the
model does not adjust for all relevant factors; second, and conversely,
included-variable bias -- a lesser-known phenomenon -- can skew results if the
set of covariates includes irrelevant factors. Here we introduce a new,
three-step statistical method, which we call risk-adjusted regression, to
address both concerns in settings where decision makers have clearly measurable
objectives. In the first step, we use all available covariates to estimate the
value, or inversely, the risk, of taking a certain action, such as approving a
loan application or hiring a job candidate. Second, we measure disparities in
decisions after adjusting for these risk estimates alone, mitigating the
problem of included-variable bias. Finally, in the third step, we assess the
sensitivity of results to potential mismeasurement of risk, addressing concerns
about omitted-variable bias. To do so, we develop a novel, non-parametric
sensitivity analysis that yields tight bounds on the true disparity in terms of
the average gap between true and estimated risk -- a single interpretable
parameter that facilitates credible estimates. We demonstrate this approach on
a detailed dataset of 2.2 million police stops of pedestrians in New York City,
and show that traditional statistical tests of discrimination can substantially
underestimate the magnitude of disparities.
