  We explore active learning (AL) for improving the accuracy of new domains in
a natural language understanding (NLU) system. We propose an algorithm called
Majority-CRF that uses an ensemble of classification models to guide the
selection of relevant utterances, as well as a sequence labeling model to help
prioritize informative examples. Experiments with three domains show that
Majority-CRF achieves 6.6%-9% relative error rate reduction compared to random
sampling with the same annotation budget, and statistically significant
improvements compared to other AL approaches. Additionally, case studies with
human-in-the-loop AL on six new domains show 4.6%-9% improvement on an existing
NLU system.
