  Quantifying variable importance is essential for answering high-stakes
questions in fields like genetics, public policy, and medicine. Current methods
generally calculate variable importance for a given model trained on a given
dataset. However, for a given dataset, there may be many models that explain
the target outcome equally well; without accounting for all possible
explanations, different researchers may arrive at many conflicting yet equally
valid conclusions given the same data. Additionally, even when accounting for
all possible explanations for a given dataset, these insights may not
generalize because not all good explanations are stable across reasonable data
perturbations. We propose a new variable importance framework that quantifies
the importance of a variable across the set of all good models and is stable
across the data distribution. Our framework is extremely flexible and can be
integrated with most existing model classes and global variable importance
metrics. We demonstrate through experiments that our framework recovers
variable importance rankings for complex simulation setups where other methods
fail. Further, we show that our framework accurately estimates the true
importance of a variable for the underlying data distribution. We provide
theoretical guarantees on the consistency and finite sample error rates for our
estimator. Finally, we demonstrate its utility with a real-world case study
exploring which genes are important for predicting HIV load in persons with
HIV, highlighting an important gene that has not previously been studied in
connection with HIV. Code is available at
https://github.com/jdonnelly36/Rashomon_Importance_Distribution.
