  Recent studies on deep ensembles have identified the sharpness of the local
minima of individual learners and the diversity of the ensemble members as key
factors in improving test-time performance. Building on this, our study
investigates the interplay between sharpness and diversity within deep
ensembles, illustrating their crucial role in robust generalization to both
in-distribution (ID) and out-of-distribution (OOD) data. We discover a
trade-off between sharpness and diversity: minimizing the sharpness in the loss
landscape tends to diminish the diversity of individual members within the
ensemble, adversely affecting the ensemble's improvement. The trade-off is
justified through our theoretical analysis and verified empirically through
extensive experiments. To address the issue of reduced diversity, we introduce
SharpBalance, a novel training approach that balances sharpness and diversity
within ensembles. Theoretically, we show that our training strategy achieves a
better sharpness-diversity trade-off. Empirically, we conducted comprehensive
evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed
that SharpBalance not only effectively improves the sharpness-diversity
trade-off, but also significantly improves ensemble performance in ID and OOD
scenarios.
