  NLP-powered automatic question generation (QG) techniques carry great
pedagogical potential of saving educators' time and benefiting student
learning. Yet, QG systems have not been widely adopted in classrooms to date.
In this work, we aim to pinpoint key impediments and investigate how to improve
the usability of automatic QG techniques for educational purposes by
understanding how instructors construct questions and identifying touch points
to enhance the underlying NLP models. We perform an in-depth need finding study
with 11 instructors across 7 different universities, and summarize their
thought processes and needs when creating questions. While instructors show
great interests in using NLP systems to support question design, none of them
has used such tools in practice. They resort to multiple sources of
information, ranging from domain knowledge to students' misconceptions, all of
which missing from today's QG systems. We argue that building effective
human-NLP collaborative QG systems that emphasize instructor control and
explainability is imperative for real-world adoption. We call for QG systems to
provide process-oriented support, use modular design, and handle diverse
sources of input.
