a human-in-the-loop system
This project aims at creating a “human-in-the-loop” system for building high-quality student affect detectors and interventions for classroom use.
Though teachers often participate in the initial design of adaptive learning systems, and learning analytics dashboards bring information on students back to teachers, the partnership between teachers and adaptive systems remains limited. Adaptive intervention often fails to leverage the insights on learning that teachers have accumulated through years of practice. Instead, we envision the future of teaching with adaptive learning technology as one in which technology actively seeks and incorporates teacher feedback into a human-in-the-loop system, a symbiosis between teachers and technology that leverages the strengths of both.
Figure 1: Our system i) suggests the teacher to conduct observations of the affective state of students ii) collect self-reported confidence ratings from teachers to improve affect detection, and iii) show teachers the detected affective states and observe how they intervene with students
Figure 2: Average AUCs on the test sets for the student affect detector on an existing ASSISTments dataset with 100 initial labeled student observations. Our active learning method outperforms every baseline.
We focus on “sensor-free” detectors of affect that use only the data students generate when they learn within computer-based learning platforms to detect their affective states. These detectors take the form of machine learning-based classifiers trained on student affective state labels provided by human observers in real classrooms.
In the first stage of the project, we will employ active learning methods to suggest to teachers that they observe students whose affective states are more informative to improving the classifier, rather than the current method of observing students in round-robin fashion. We will explore whether these more informative observations can improve affect detection with less teacher effort and no disruption to classes. In the second stage of the project, we will incorporate richer data on how certain teachers judge student affect into the classifiers to improve their quality and degree of alignment to teacher judgements, increasing their usefulness to teachers. In the third stage of the project, we will use crowdsourcing to solicit ideas from teachers as to when specific affect interventions will be appropriate for specific students, and will build these ideas into a comprehensive intervention system. We will test this intervention system in an experimental study in real classrooms.
Figure 3: The teacher interface to identify student affective states.