  The ability to detect failures and anomalies are fundamental requirements for
building reliable systems for computer vision applications, especially
safety-critical applications of semantic segmentation, such as autonomous
driving and medical image analysis. In this paper, we systematically study
failure and anomaly detection for semantic segmentation and propose a unified
framework, consisting of two modules, to address these two related problems.
The first module is an image synthesis module, which generates a synthesized
image from a segmentation layout map, and the second is a comparison module,
which computes the difference between the synthesized image and the input
image. We validate our framework on three challenging datasets and improve the
state-of-the-arts by large margins, \emph{i.e.}, 6% AUPR-Error on Cityscapes,
7% Pearson correlation on pancreatic tumor segmentation in MSD and 20% AUPR on
StreetHazards anomaly segmentation.
