  Deep neural network, despite its remarkable capability of discriminating
targeted in-distribution samples, shows poor performance on detecting anomalous
out-of-distribution data. To address this defect, state-of-the-art solutions
choose to train deep networks on an auxiliary dataset of outliers. Various
training criteria for these auxiliary outliers are proposed based on heuristic
intuitions. However, we find that these intuitively designed outlier training
criteria can hurt in-distribution learning and eventually lead to inferior
performance. To this end, we identify three causes of the in-distribution
incompatibility: contradictory gradient, false likelihood, and distribution
shift. Based on our new understandings, we propose a new out-of-distribution
detection method by adapting both the top-design of deep models and the loss
function. Our method achieves in-distribution compatibility by pursuing less
interference with the probabilistic characteristic of in-distribution features.
On several benchmarks, our method not only achieves the state-of-the-art
out-of-distribution detection performance but also improves the in-distribution
accuracy.
