  While several types of post hoc explanation methods have been proposed in
recent literature, there is very little work on systematically benchmarking
these methods. Here, we introduce OpenXAI, a comprehensive and extensible
open-source framework for evaluating and benchmarking post hoc explanation
methods. OpenXAI comprises of the following key components: (i) a flexible
synthetic data generator and a collection of diverse real-world datasets,
pre-trained models, and state-of-the-art feature attribution methods, and (ii)
open-source implementations of eleven quantitative metrics for evaluating
faithfulness, stability (robustness), and fairness of explanation methods, in
turn providing comparisons of several explanation methods across a wide variety
of metrics, models, and datasets. OpenXAI is easily extensible, as users can
readily evaluate custom explanation methods and incorporate them into our
leaderboards. Overall, OpenXAI provides an automated end-to-end pipeline that
not only simplifies and standardizes the evaluation of post hoc explanation
methods, but also promotes transparency and reproducibility in benchmarking
these methods. While the first release of OpenXAI supports only tabular
datasets, the explanation methods and metrics that we consider are general
enough to be applicable to other data modalities. OpenXAI datasets and models,
implementations of state-of-the-art explanation methods and evaluation metrics,
are publicly available at this GitHub link.
