  Visual anomaly detection aims to learn normality from normal images, but
existing approaches are fragmented across various tasks: defect detection,
semantic anomaly detection, multi-class anomaly detection, and anomaly
clustering. This one-task-one-model approach is resource-intensive and incurs
high maintenance costs as the number of tasks increases. We present UniFormaly,
a universal and powerful anomaly detection framework. We emphasize the
necessity of our off-the-shelf approach by pointing out a suboptimal issue in
online encoder-based methods. We introduce Back Patch Masking (BPM) and top
k-ratio feature matching to achieve unified anomaly detection. BPM eliminates
irrelevant background regions using a self-attention map from self-supervised
ViTs. This operates in a task-agnostic manner and alleviates memory storage
consumption, scaling to tasks with large-scale datasets. Top k-ratio feature
matching unifies anomaly levels and tasks by casting anomaly scoring into
multiple instance learning. Finally, UniFormaly achieves outstanding results on
various tasks and datasets. Codes are available at
https://github.com/YoojLee/Uniformaly.
