  Anomaly detection requires detecting abnormal samples in large unlabeled
datasets. While progress in deep learning and the advent of foundation models
has produced powerful zero-shot anomaly detection methods, their deployment in
practice is often hindered by the lack of labeled data -- without it, their
detection performance cannot be evaluated reliably. In this work, we propose
SWSA (Selection With Synthetic Anomalies): a general-purpose framework to
select image-based anomaly detectors with a generated synthetic validation set.
Our proposed anomaly generation method assumes access to only a small support
set of normal images and requires no training or fine-tuning. Once generated,
our synthetic validation set is used to create detection tasks that compose a
validation framework for model selection. In an empirical study, we find that
SWSA often selects models that match selections made with a ground-truth
validation set, resulting in higher AUROCs than baseline methods. We also find
that SWSA selects prompts for CLIP-based anomaly detection that outperform
baseline prompt selection strategies on all datasets, including the challenging
MVTec-AD and VisA datasets.
