  We provide a new information-theoretic generalization error bound that is
exactly tight (i.e., matching even the constant) for the canonical quadratic
Gaussian (location) problem. Most existing bounds are order-wise loose in this
setting, which has raised concerns about the fundamental capability of
information-theoretic bounds in reasoning the generalization behavior for
machine learning. The proposed new bound adopts the individual-sample-based
approach proposed by Bu et al., but also has several key new ingredients.
Firstly, instead of applying the change of measure inequality on the loss
function, we apply it to the generalization error function itself; secondly,
the bound is derived in a conditional manner; lastly, a reference distribution
is introduced. The combination of these components produces a
KL-divergence-based generalization error bound. We show that although the
latter two new ingredients can help make the bound exactly tight, removing them
does not significantly degrade the bound, leading to an asymptotically tight
mutual-information-based bound. We further consider the vector Gaussian
setting, where a direct application of the proposed bound again does not lead
to tight bounds except in special cases. A refined bound is then proposed for
decomposable loss functions, leading to a tight bound for the vector setting.
