  Supervised learning of every possible pathology is unrealistic for many
primary care applications like health screening. Image anomaly detection
methods that learn normal appearance from only healthy data have shown
promising results recently. We propose an alternative to image
reconstruction-based and image embedding-based methods and propose a new
self-supervised method to tackle pathological anomaly detection. Our approach
originates in the foreign patch interpolation (FPI) strategy that has shown
superior performance on brain MRI and abdominal CT data. We propose to use a
better patch interpolation strategy, Poisson image interpolation (PII), which
makes our method suitable for applications in challenging data regimes. PII
outperforms state-of-the-art methods by a good margin when tested on surrogate
tasks like identifying common lung anomalies in chest X-rays or hypo-plastic
left heart syndrome in prenatal, fetal cardiac ultrasound images. Code
available at https://github.com/jemtan/PII.
