  Despite agreement on the importance of detecting out-of-distribution (OOD)
examples, there is little consensus on the formal definition of OOD examples
and how to best detect them. We categorize these examples by whether they
exhibit a background shift or a semantic shift, and find that the two major
approaches to OOD detection, model calibration and density estimation (language
modeling for text), have distinct behavior on these types of OOD data. Across
14 pairs of in-distribution and OOD English natural language understanding
datasets, we find that density estimation methods consistently beat calibration
methods in background shift settings, while performing worse in semantic shift
settings. In addition, we find that both methods generally fail to detect
examples from challenge data, highlighting a weak spot for current methods.
Since no single method works well across all settings, our results call for an
explicit definition of OOD examples when evaluating different detection
methods.
