  Autoencoder, as an essential part of many anomaly detection methods, is
lacking flexibility on normal data in complex datasets. U-Net is proved to be
effective for this purpose but overfits on the training data if trained by just
using reconstruction error similar to other AE-based frameworks.
Puzzle-solving, as a pretext task of self-supervised learning (SSL) methods,
has earlier proved its ability in learning semantically meaningful features. We
show that training U-Nets based on this task is an effective remedy that
prevents overfitting and facilitates learning beyond pixel-level features.
Shortcut solutions, however, are a big challenge in SSL tasks, including jigsaw
puzzles. We propose adversarial robust training as an effective automatic
shortcut removal. We achieve competitive or superior results compared to the
State of the Art (SOTA) anomaly detection methods on various toy and real-world
datasets. Unlike many competitors, the proposed framework is stable, fast,
data-efficient, and does not require unprincipled early stopping.
