  We encounter variables with little variation often in educational data mining
(EDM) due to the demographics of higher education and the questions we ask.
Yet, little work has examined how to analyze such data. Therefore, we conducted
a simulation study using logistic regression, penalized regression, and random
forest. We systematically varied the fraction of positive outcomes, feature
imbalances, and odds ratios. We find the algorithms treat features with the
same odds ratios differently based on the features' imbalance and the outcome
imbalance. While none of the algorithms fully solved how to handle imbalanced
data, penalized approaches such as Firth and Log-F reduced the difference
between the built-in odds ratio and value determined by the algorithm. Our
results suggest that EDM studies might contain false negatives when determining
which variables are related to an outcome. We then apply our findings to a
graduate admissions data set. We end by proposing recommendations that
researchers should consider penalized regression for data sets on the order of
hundreds of cases and should include more context about their data in
publications such as the outcome and feature imbalances.
