  This paper explores the problem of Generalist Anomaly Detection (GAD), aiming
to train one single detection model that can generalize to detect anomalies in
diverse datasets from different application domains without any further
training on the target data. Some recent studies have shown that large
pre-trained Visual-Language Models (VLMs) like CLIP have strong generalization
capabilities on detecting industrial defects from various datasets, but their
methods rely heavily on handcrafted text prompts about defects, making them
difficult to generalize to anomalies in other applications, e.g., medical image
anomalies or semantic anomalies in natural images. In this work, we propose to
train a GAD model with few-shot normal images as sample prompts for AD on
diverse datasets on the fly. To this end, we introduce a novel approach that
learns an in-context residual learning model for GAD, termed InCTRL. It is
trained on an auxiliary dataset to discriminate anomalies from normal samples
based on a holistic evaluation of the residuals between query images and
few-shot normal sample prompts. Regardless of the datasets, per definition of
anomaly, larger residuals are expected for anomalies than normal samples,
thereby enabling InCTRL to generalize across different domains without further
training. Comprehensive experiments on nine AD datasets are performed to
establish a GAD benchmark that encapsulate the detection of industrial defect
anomalies, medical anomalies, and semantic anomalies in both one-vs-all and
multi-class setting, on which InCTRL is the best performer and significantly
outperforms state-of-the-art competing methods. Code is available at
https://github.com/mala-lab/InCTRL.
