  Even though auto-encoders (AEs) have the desirable property of learning
compact representations without labels and have been widely applied to
out-of-distribution (OoD) detection, they are generally still poorly understood
and are used incorrectly in detecting outliers where the normal and abnormal
distributions are strongly overlapping. In general, the learned manifold is
assumed to contain key information that is only important for describing
samples within the training distribution, and that the reconstruction of
outliers leads to high residual errors. However, recent work suggests that AEs
are likely to be even better at reconstructing some types of OoD samples. In
this work, we challenge this assumption and investigate what auto-encoders
actually learn when they are posed to solve two different tasks. First, we
propose two metrics based on the Fr\'echet inception distance (FID) and
confidence scores of a trained classifier to assess whether AEs can learn the
training distribution and reliably recognize samples from other domains.
Second, we investigate whether AEs are able to synthesize normal images from
samples with abnormal regions, on a more challenging lung pathology detection
task. We have found that state-of-the-art (SOTA) AEs are either unable to
constrain the latent manifold and allow reconstruction of abnormal patterns, or
they are failing to accurately restore the inputs from their latent
distribution, resulting in blurred or misaligned reconstructions. We propose
novel deformable auto-encoders (MorphAEus) to learn perceptually aware global
image priors and locally adapt their morphometry based on estimated dense
deformation fields. We demonstrate superior performance over unsupervised
methods in detecting OoD and pathology.
