  Detecting anomalies in musculoskeletal radiographs is of paramount importance
for large-scale screening in the radiology workflow. Supervised deep networks
take for granted a large number of annotations by radiologists, which is often
prohibitively very time-consuming to acquire. Moreover, supervised systems are
tailored to closed set scenarios, e.g., trained models suffer from overfitting
to previously seen rare anomalies at training. Instead, our approach's
rationale is to use task agnostic pretext tasks to leverage unlabeled data
based on a cross-sample similarity measure. Besides, we formulate a complex
distribution of data from normal class within our framework to avoid a
potential bias on the side of anomalies. Through extensive experiments, we show
that our method outperforms baselines across unsupervised and self-supervised
anomaly detection settings on a real-world medical dataset, the MURA dataset.
We also provide rich ablation studies to analyze each training stage's effect
and loss terms on the final performance.
