  Ensembles of deep neural networks demonstrate improved performance over
single models. For enhancing the diversity of ensemble members while keeping
their performance, particle-based inference methods offer a promising approach
from a Bayesian perspective. However, the best way to apply these methods to
neural networks is still unclear: seeking samples from the weight-space
posterior suffers from inefficiency due to the over-parameterization issues,
while seeking samples directly from the function-space posterior often results
in serious underfitting. In this study, we propose optimizing particles in the
feature space where the activation of a specific intermediate layer lies to
address the above-mentioned difficulties. Our method encourages each member to
capture distinct features, which is expected to improve ensemble prediction
robustness. Extensive evaluation on real-world datasets shows that our model
significantly outperforms the gold-standard Deep Ensembles on various metrics,
including accuracy, calibration, and robustness. Code is available at
https://github.com/DensoITLab/featurePI .
