  Despite significant advances in image anomaly detection and segmentation, few
methods use 3D information. We utilize a recently introduced 3D anomaly
detection dataset to evaluate whether or not using 3D information is a lost
opportunity. First, we present a surprising finding: standard color-only
methods outperform all current methods that are explicitly designed to exploit
3D information. This is counter-intuitive as even a simple inspection of the
dataset shows that color-only methods are insufficient for images containing
geometric anomalies. This motivates the question: how can anomaly detection
methods effectively use 3D information? We investigate a range of shape
representations including hand-crafted and deep-learning-based; we demonstrate
that rotation invariance plays the leading role in the performance. We uncover
a simple 3D-only method that beats all recent approaches while not using deep
learning, external pre-training datasets, or color information. As the 3D-only
method cannot detect color and texture anomalies, we combine it with
color-based features, significantly outperforming previous state-of-the-art.
Our method, dubbed BTF (Back to the Feature) achieves pixel-wise ROCAUC: 99.3%
and PRO: 96.4% on MVTec 3D-AD.
