  In order to obtain reliable accuracy estimates for automatic MOOC dropout
predictors, it is important to train and test them in a manner consistent with
how they will be used in practice. Yet most prior research on MOOC dropout
prediction has measured test accuracy on the same course used for training the
classifier, which can lead to overly optimistic accuracy estimates. In order to
understand better how accuracy is affected by the training+testing regime, we
compared the accuracy of a standard dropout prediction architecture
(clickstream features + logistic regression) across 4 different training
paradigms. Results suggest that (1) training and testing on the same course
("post-hoc") can overestimate accuracy by several percentage points; (2)
dropout classifiers trained on proxy labels based on students' persistence are
surprisingly competitive with post-hoc training (87.33% versus 90.20% AUC
averaged over 8 weeks of 40 HarvardX MOOCs); and (3) classifier performance
does not vary significantly with the academic discipline. Finally, we also
research new dropout prediction architectures based on deep, fully-connected,
feed-forward neural networks and find that (4) networks with as many as 5
hidden layers can statistically significantly increase test accuracy over that
of logistic regression.
