  Predictive student models are increasingly used in learning environments due
to their ability to enhance educational outcomes and support stakeholders in
making informed decisions. However, predictive models can be biased and produce
unfair outcomes, leading to potential discrimination against some students and
possible harmful long-term implications. This has prompted research on fairness
metrics meant to capture and quantify such biases. Nonetheless, so far,
existing fairness metrics used in education are predictive
performance-oriented, focusing on assessing biased outcomes across groups of
students, without considering the behaviors of the models nor the severity of
the biases in the outcomes. Therefore, we propose a novel metric, the Model
Absolute Density Distance (MADD), to analyze models' discriminatory behaviors
independently from their predictive performance. We also provide a
complementary visualization-based analysis to enable fine-grained human
assessment of how the models discriminate between groups of students. We
evaluate our approach on the common task of predicting student success in
online courses, using several common predictive classification models on an
open educational dataset. We also compare our metric to the only predictive
performance-oriented fairness metric developed in education, ABROCA. Results on
this dataset show that: (1) fair predictive performance does not guarantee fair
models' behaviors and thus fair outcomes, (2) there is no direct relationship
between data bias and predictive performance bias nor discriminatory behaviors
bias, and (3) trained on the same data, models exhibit different discriminatory
behaviors, according to different sensitive features too. We thus recommend
using the MADD on models that show satisfying predictive performance, to gain a
finer-grained understanding on how they behave and to refine models selection
and their usage.
