  Out-of-distribution (OOD) detection is an indispensable aspect of secure AI
when deploying machine learning models in real-world applications. Previous
paradigms either explore better scoring functions or utilize the knowledge of
outliers to equip the models with the ability of OOD detection. However, few of
them pay attention to the intrinsic OOD detection capability of the given
model. In this work, we generally discover the existence of an intermediate
stage of a model trained on in-distribution (ID) data having higher OOD
detection performance than that of its final stage across different settings,
and further identify one critical data-level attribution to be learning with
the atypical samples. Based on such insights, we propose a novel method,
Unleashing Mask, which aims to restore the OOD discriminative capabilities of
the well-trained model with ID data. Our method utilizes a mask to figure out
the memorized atypical samples, and then finetune the model or prune it with
the introduced mask to forget them. Extensive experiments and analysis
demonstrate the effectiveness of our method. The code is available at:
https://github.com/tmlr-group/Unleashing-Mask.
