  This Research Full Paper explores automatic identification of ineffective
learning questions in the context of large-scale computer science classes. The
immediate and accurate identification of ineffective learning questions opens
the door to possible automated facilitation on a large scale, such as alerting
learners to revise questions and providing adaptive question revision
suggestions. To achieve this, 983 questions were collected from a question &
answer platform implemented by an introductory programming course over three
semesters in a large research university in the Southeastern United States.
Questions were firstly manually classified into three hierarchical categories:
1) learning-irrelevant questions, 2) effective learning-relevant questions, 3)
ineffective learningrelevant questions. The inter-rater reliability of the
manual classification (Cohen's Kappa) was .88. Four different machine learning
algorithms were then used to automatically classify the questions, including
Naive Bayes Multinomial, Logistic Regression, Support Vector Machines, and
Boosted Decision Tree. Both flat and single path strategies were explored, and
the most effective algorithms under both strategies were identified and
discussed. This study contributes to the automatic determination of learning
question quality in computer science, and provides evidence for the feasibility
of automated facilitation of online question & answer in large scale computer
science classes.
