  Unsupervised learning can leverage large-scale data sources without the need
for annotations. In this context, deep learning-based auto encoders have shown
great potential in detecting anomalies in medical images. However,
state-of-the-art anomaly scores are still based on the reconstruction error,
which lacks in two essential parts: it ignores the model-internal
representation employed for reconstruction, and it lacks formal assertions and
comparability between samples. We address these shortcomings by proposing the
Context-encoding Variational Autoencoder (ceVAE) which combines reconstruction-
with density-based anomaly scoring. This improves the sample- as well as
pixel-wise results. In our experiments on the BraTS-2017 and ISLES-2015
segmentation benchmarks, the ceVAE achieves unsupervised ROC-AUCs of 0.95 and
0.89, respectively, thus outperforming state-of-the-art methods by a
considerable margin.
