  Large-scale administrative data is a common input in early warning systems
for college dropout in higher education. Still, the terminology and methodology
vary significantly across existing studies, and the implications of different
modeling decisions are not fully understood. This study provides a systematic
evaluation of contributing factors and predictive performance of machine
learning models over time and across different student groups. Drawing on
twelve years of administrative data at a large public university in the US, we
find that dropout prediction at the end of the second year has a 20% higher AUC
than at the time of enrollment in a Random Forest model. Also, most predictive
factors at the time of enrollment, including demographics and high school
performance, are quickly superseded in predictive importance by college
performance and in later stages by enrollment behavior. Regarding variability
across student groups, college GPA has more predictive value for students from
traditionally disadvantaged backgrounds than their peers. These results can
help researchers and administrators understand the comparative value of
different data sources when building early warning systems and optimizing
decisions under specific policy goals.
