  Information Bottlenecks (IBs) learn representations that generalize to unseen
data by information compression. However, existing IBs are practically unable
to guarantee generalization in real-world scenarios due to the vacuous
generalization bound. The recent PAC-Bayes IB uses information complexity
instead of information compression to establish a connection with the mutual
information generalization bound. However, it requires the computation of
expensive second-order curvature, which hinders its practical application. In
this paper, we establish the connection between the recognizability of
representations and the recent functional conditional mutual information
(f-CMI) generalization bound, which is significantly easier to estimate. On
this basis we propose a Recognizable Information Bottleneck (RIB) which
regularizes the recognizability of representations through a recognizability
critic optimized by density ratio matching under the Bregman divergence.
Extensive experiments on several commonly used datasets demonstrate the
effectiveness of the proposed method in regularizing the model and estimating
the generalization gap.
