  One of the unsolved challenges in the field of Explainable AI (XAI) is
determining how to most reliably estimate the quality of an explanation method
in the absence of ground truth explanation labels. Resolving this issue is of
utmost importance as the evaluation outcomes generated by competing evaluation
methods (or ''quality estimators''), which aim at measuring the same property
of an explanation method, frequently present conflicting rankings. Such
disagreements can be challenging for practitioners to interpret, thereby
complicating their ability to select the best-performing explanation method. We
address this problem through a meta-evaluation of different quality estimators
in XAI, which we define as ''the process of evaluating the evaluation method''.
Our novel framework, MetaQuantus, analyses two complementary performance
characteristics of a quality estimator: its resilience to noise and reactivity
to randomness, thus circumventing the need for ground truth labels. We
demonstrate the effectiveness of our framework through a series of experiments,
targeting various open questions in XAI such as the selection and
hyperparameter optimisation of quality estimators. Our work is released under
an open-source license (https://github.com/annahedstroem/MetaQuantus) to serve
as a development tool for XAI- and Machine Learning (ML) practitioners to
verify and benchmark newly constructed quality estimators in a given
explainability context. With this work, we provide the community with clear and
theoretically-grounded guidance for identifying reliable evaluation methods,
thus facilitating reproducibility in the field.
