  In this work, we aim to address the challenging task of open set recognition
(OSR). Many recent OSR methods rely on auto-encoders to extract class-specific
features by a reconstruction strategy, requiring the network to restore the
input image on pixel-level. This strategy is commonly over-demanding for OSR
since class-specific features are generally contained in target objects, not in
all pixels. To address this shortcoming, here we discard the pixel-level
reconstruction strategy and pay more attention to improving the effectiveness
of class-specific feature extraction. We propose a mutual information-based
method with a streamlined architecture, Maximal Mutual Information Open Set
Recognition (M2IOSR). The proposed M2IOSR only uses an encoder to extract
class-specific features by maximizing the mutual information between the given
input and its latent features across multiple scales. Meanwhile, to further
reduce the open space risk, latent features are constrained to class
conditional Gaussian distributions by a KL-divergence loss function. In this
way, a strong function is learned to prevent the network from mapping different
observations to similar latent features and help the network extract
class-specific features with desired statistical characteristics. The proposed
method significantly improves the performance of baselines and achieves new
state-of-the-art results on several benchmarks consistently.
