  Predictive multiplicity occurs when classification models with statistically
indistinguishable performances assign conflicting predictions to individual
samples. When used for decision-making in applications of consequence (e.g.,
lending, education, criminal justice), models developed without regard for
predictive multiplicity may result in unjustified and arbitrary decisions for
specific individuals. We introduce a new metric, called Rashomon Capacity, to
measure predictive multiplicity in probabilistic classification. Prior metrics
for predictive multiplicity focus on classifiers that output thresholded (i.e.,
0-1) predicted classes. In contrast, Rashomon Capacity applies to probabilistic
classifiers, capturing more nuanced score variations for individual samples. We
provide a rigorous derivation for Rashomon Capacity, argue its intuitive
appeal, and demonstrate how to estimate it in practice. We show that Rashomon
Capacity yields principled strategies for disclosing conflicting models to
stakeholders. Our numerical experiments illustrate how Rashomon Capacity
captures predictive multiplicity in various datasets and learning models,
including neural networks. The tools introduced in this paper can help data
scientists measure and report predictive multiplicity prior to model
deployment.
