  Most language understanding models in task-oriented dialog systems are
trained on a small amount of annotated training data, and evaluated in a small
set from the same distribution. However, these models can lead to system
failure or undesirable output when being exposed to natural language
perturbation or variation in practice. In this paper, we conduct comprehensive
evaluation and analysis with respect to the robustness of natural language
understanding models, and introduce three important aspects related to language
understanding in real-world dialog systems, namely, language variety, speech
characteristics, and noise perturbation. We propose a model-agnostic toolkit
LAUG to approximate natural language perturbations for testing the robustness
issues in task-oriented dialog. Four data augmentation approaches covering the
three aspects are assembled in LAUG, which reveals critical robustness issues
in state-of-the-art models. The augmented dataset through LAUG can be used to
facilitate future research on the robustness testing of language understanding
in task-oriented dialog.
