  This paper presents a novel evaluation framework for Out-of-Distribution
(OOD) detection that aims to assess the performance of machine learning models
in more realistic settings. We observed that the real-world requirements for
testing OOD detection methods are not satisfied by the current testing
protocols. They usually encourage methods to have a strong bias towards a low
level of diversity in normal data. To address this limitation, we propose new
OOD test datasets (CIFAR-10-R, CIFAR-100-R, and ImageNet-30-R) that can allow
researchers to benchmark OOD detection performance under realistic distribution
shifts. Additionally, we introduce a Generalizability Score (GS) to measure the
generalization ability of a model during OOD detection. Our experiments
demonstrate that improving the performance on existing benchmark datasets does
not necessarily improve the usability of OOD detection models in real-world
scenarios. While leveraging deep pre-trained features has been identified as a
promising avenue for OOD detection research, our experiments show that
state-of-the-art pre-trained models tested on our proposed datasets suffer a
significant drop in performance. To address this issue, we propose a
post-processing stage for adapting pre-trained features under these
distribution shifts before calculating the OOD scores, which significantly
enhances the performance of state-of-the-art pre-trained models on our
benchmarks.
