  Anomaly segmentation is a crucial task for safety-critical applications, such
as autonomous driving in urban scenes, where the goal is to detect
out-of-distribution (OOD) objects with categories which are unseen during
training. The core challenge of this task is how to distinguish hard
in-distribution samples from OOD samples, which has not been explicitly
discussed yet. In this paper, we propose a novel and simple approach named
Consensus Synergizes with Memory (CosMe) to address this challenge, inspired by
the psychology finding that groups perform better than individuals on memory
tasks. The main idea is 1) building a memory bank which consists of seen
prototypes extracted from multiple layers of the pre-trained segmentation model
and 2) training an auxiliary model that mimics the behavior of the pre-trained
model, and then measuring the consensus of their mid-level features as
complementary cues that synergize with the memory bank. CosMe is good at
distinguishing between hard in-distribution examples and OOD samples.
Experimental results on several urban scene anomaly segmentation datasets show
that CosMe outperforms previous approaches by large margins.
