  Unsupervised anomaly detection methods offer a promising and flexible
alternative to supervised approaches, holding the potential to revolutionize
medical scan analysis and enhance diagnostic performance.
  In the current landscape, it is commonly assumed that differences between a
test case and the training distribution are attributed solely to pathological
conditions, implying that any disparity indicates an anomaly. However, the
presence of other potential sources of distributional shift, including scanner,
age, sex, or race, is frequently overlooked. These shifts can significantly
impact the accuracy of the anomaly detection task. Prominent instances of such
failures have sparked concerns regarding the bias, credibility, and fairness of
anomaly detection.
  This work presents a novel analysis of biases in unsupervised anomaly
detection. By examining potential non-pathological distributional shifts
between the training and testing distributions, we shed light on the extent of
these biases and their influence on anomaly detection results. Moreover, this
study examines the algorithmic limitations that arise due to biases, providing
valuable insights into the challenges encountered by anomaly detection
algorithms in accurately learning and capturing the entire range of variability
present in the normative distribution. Through this analysis, we aim to enhance
the understanding of these biases and pave the way for future improvements in
the field. Here, we specifically investigate Alzheimer's disease detection from
brain MR imaging as a case study, revealing significant biases related to sex,
race, and scanner variations that substantially impact the results. These
findings align with the broader goal of improving the reliability, fairness,
and effectiveness of anomaly detection in medical imaging.
