  This work addresses the challenge of providing consistent explanations for
predictive models in the presence of model indeterminacy, which arises due to
the existence of multiple (nearly) equally well-performing models for a given
dataset and task. Despite their similar performance, such models often exhibit
inconsistent or even contradictory explanations for their predictions, posing
challenges to end users who rely on these models to make critical decisions.
Recognizing this issue, we introduce ensemble methods as an approach to enhance
the consistency of the explanations provided in these scenarios. Leveraging
insights from recent work on neural network loss landscapes and mode
connectivity, we devise ensemble strategies to efficiently explore the
underspecification set -- the set of models with performance variations
resulting solely from changes in the random seed during training. Experiments
on five benchmark financial datasets reveal that ensembling can yield
significant improvements when it comes to explanation similarity, and
demonstrate the potential of existing ensemble methods to explore the
underspecification set efficiently. Our findings highlight the importance of
considering model indeterminacy when interpreting explanations and showcase the
effectiveness of ensembles in enhancing the reliability of explanations in
machine learning.
