  Deep Learning models perform unreliably when the data comes from a
distribution different from the training one. In critical applications such as
medical imaging, out-of-distribution (OOD) detection methods help to identify
such data samples, preventing erroneous predictions. In this paper, we further
investigate the OOD detection effectiveness when applied to 3D medical image
segmentation. We design several OOD challenges representing clinically
occurring cases and show that none of these methods achieve acceptable
performance. Methods not dedicated to segmentation severely fail to perform in
the designed setups; their best mean false positive rate at 95% true positive
rate (FPR) is 0.59. Segmentation-dedicated ones still achieve suboptimal
performance, with the best mean FPR of 0.31 (lower is better). To indicate this
suboptimality, we develop a simple method called Intensity Histogram Features
(IHF), which performs comparable or better in the same challenges, with a mean
FPR of 0.25. Our findings highlight the limitations of the existing OOD
detection methods on 3D medical images and present a promising avenue for
improving them. To facilitate research in this area, we release the designed
challenges as a publicly available benchmark and formulate practical criteria
to test the OOD detection generalization beyond the suggested benchmark. We
also propose IHF as a solid baseline to contest the emerging methods.
