  Machine learning algorithms have recently been used to predict students'
performance in an introductory physics class. The prediction model classified
students as those likely to receive an A or B or students likely to receive a
grade of C, D, F or withdraw from the class. Early prediction could better
allow the direction of educational interventions and the allocation of
educational resources. However, the performance metrics used in that study
become unreliable when used to classify whether a student would receive an A, B
or C (the ABC outcome) or if they would receive a D, F or withdraw (W) from the
class (the DFW outcome) because the outcome is substantially unbalanced with
between 10\% to 20\% of the students receiving a D, F, or W. This work presents
techniques to adjust the prediction models and alternate model performance
metrics more appropriate for unbalanced outcome variables. These techniques
were applied to three samples drawn from introductory mechanics classes at two
institutions ($N=7184$, $1683$, and $926$). Applying the same methods as the
earlier study produced a classifier that was very inaccurate, classifying only
16\% of the DFW cases correctly; tuning the model increased the DFW
classification accuracy to 43\%. Using a combination of institutional and
in-class data improved DFW accuracy to 53\% by the second week of class. As in
the prior study, demographic variables such as gender, underrepresented
minority status, first-generation college student status, and low socioeconomic
status were not important variables in the final prediction models.
