  Novelty detection is the process of determining whether a query example
differs from the learned training distribution. Previous methods attempt to
learn the representation of the normal samples via generative adversarial
networks (GANs). However, they will suffer from instability training, mode
dropping, and low discriminative ability. Recently, various pretext tasks (e.g.
rotation prediction and clustering) have been proposed for self-supervised
learning in novelty detection. However, the learned latent features are still
low discriminative. We overcome such problems by introducing a novel
decoder-encoder framework. Firstly, a generative network (a.k.a. decoder)
learns the representation by mapping the initialized latent vector to an image.
In particular, this vector is initialized by considering the entire
distribution of training data to avoid the problem of mode-dropping. Secondly,
a contrastive network (a.k.a. encoder) aims to ``learn to compare'' through
mutual information estimation, which directly helps the generative network to
obtain a more discriminative representation by using a negative data
augmentation strategy. Extensive experiments show that our model has
significant superiority over cutting-edge novelty detectors and achieves new
state-of-the-art results on some novelty detection benchmarks, e.g. CIFAR10 and
DCASE. Moreover, our model is more stable for training in a non-adversarial
manner, compared to other adversarial based novelty detection methods.
