  The Centers for Disease Control and Prevention (CDC) coordinates a
labor-intensive process to measure the prevalence of autism spectrum disorder
(ASD) among children in the United States. Random forests methods have shown
promise in speeding up this process, but they lag behind human classification
accuracy by about 5%. We explore whether more recently available document
classification algorithms can close this gap. We applied 8 supervised learning
algorithms to predict whether children meet the case definition for ASD based
solely on the words in their evaluations. We compared the algorithms'
performance across 10 random train-test splits of the data, using
classification accuracy, F1 score, and number of positive calls to evaluate
their potential use for surveillance. Across the 10 train-test cycles, the
random forest and support vector machine with Naive Bayes features (NB-SVM)
each achieved slightly more than 87% mean accuracy. The NB-SVM produced
significantly more false negatives than false positives (P = 0.027), but the
random forest did not, making its prevalence estimates very close to the true
prevalence in the data. The best-performing neural network performed similarly
to the random forest on both measures. The random forest performed as well as
more recently available models like the NB-SVM and the neural network, and it
also produced good prevalence estimates. NB-SVM may not be a good candidate for
use in a fully-automated surveillance workflow due to increased false
negatives. More sophisticated algorithms, like hierarchical convolutional
neural networks, may not be feasible to train due to characteristics of the
data. Current algorithms might perform better if the data are abstracted and
processed differently and if they take into account information about the
children in addition to their evaluations.
