  Colleges and universities use predictive analytics in a variety of ways to
increase student success rates. Despite the potential for predictive analytics,
two major barriers exist to their adoption in higher education: (a) the lack of
democratization in deployment, and (b) the potential to exacerbate
inequalities. Education researchers and policymakers encounter numerous
challenges in deploying predictive modeling in practice. These challenges
present in different steps of modeling including data preparation, model
development, and evaluation. Nevertheless, each of these steps can introduce
additional bias to the system if not appropriately performed. Most large-scale
and nationally representative education data sets suffer from a significant
number of incomplete responses from the research participants. While many
education-related studies addressed the challenges of missing data, little is
known about the impact of handling missing values on the fairness of predictive
outcomes in practice. In this paper, we set out to first assess the disparities
in predictive modeling outcomes for college-student success, then investigate
the impact of imputation techniques on the model performance and fairness using
a commonly used set of metrics. We conduct a prospective evaluation to provide
a less biased estimation of future performance and fairness than an evaluation
of historical data. Our comprehensive analysis of a real large-scale education
dataset reveals key insights on modeling disparities and how imputation
techniques impact the fairness of the student-success predictive outcome under
different testing scenarios. Our results indicate that imputation introduces
bias if the testing set follows the historical distribution. However, if the
injustice in society is addressed and consequently the upcoming batch of
observations is equalized, the model would be less biased.
