  Neural dialog models often lack robustness to anomalous user input and
produce inappropriate responses which leads to frustrating user experience.
Although there are a set of prior approaches to out-of-domain (OOD) utterance
detection, they share a few restrictions: they rely on OOD data or multiple
sub-domains, and their OOD detection is context-independent which leads to
suboptimal performance in a dialog. The goal of this paper is to propose a
novel OOD detection method that does not require OOD data by utilizing
counterfeit OOD turns in the context of a dialog. For the sake of fostering
further research, we also release new dialog datasets which are 3 publicly
available dialog corpora augmented with OOD turns in a controllable way. Our
method outperforms state-of-the-art dialog models equipped with a conventional
OOD detection mechanism by a large margin in the presence of OOD utterances.
