  Ensembles of independently trained neural networks are a state-of-the-art
approach to estimate predictive uncertainty in Deep Learning, and can be
interpreted as an approximation of the posterior distribution via a mixture of
delta functions. The training of ensembles relies on non-convexity of the loss
landscape and random initialization of their individual members, making the
resulting posterior approximation uncontrolled. This paper proposes a novel and
principled method to tackle this limitation, minimizing an $f$-divergence
between the true posterior and a kernel density estimator (KDE) in a function
space. We analyze this objective from a combinatorial point of view, and show
that it is submodular with respect to mixture components for any $f$.
Subsequently, we consider the problem of greedy ensemble construction. From the
marginal gain on the negative $f$-divergence, which quantifies an improvement
in posterior approximation yielded by adding a new component into the KDE, we
derive a novel diversity term for ensemble methods. The performance of our
approach is demonstrated on computer vision out-of-distribution detection
benchmarks in a range of architectures trained on multiple datasets. The source
code of our method is made publicly available at
https://github.com/Oulu-IMEDS/greedy_ensembles_training.
