  Counterfactual explanations provide means for prescriptive model explanations
by suggesting actionable feature changes (e.g., increase income) that allow
individuals to achieve favorable outcomes in the future (e.g., insurance
approval). Choosing an appropriate method is a crucial aspect for meaningful
counterfactual explanations. As documented in recent reviews, there exists a
quickly growing literature with available methods. Yet, in the absence of
widely available opensource implementations, the decision in favor of certain
models is primarily based on what is readily available. Going forward - to
guarantee meaningful comparisons across explanation methods - we present CARLA
(Counterfactual And Recourse LibrAry), a python library for benchmarking
counterfactual explanation methods across both different data sets and
different machine learning models. In summary, our work provides the following
contributions: (i) an extensive benchmark of 11 popular counterfactual
explanation methods, (ii) a benchmarking framework for research on future
counterfactual explanation methods, and (iii) a standardized set of integrated
evaluation measures and data sets for transparent and extensive comparisons of
these methods. We have open-sourced CARLA and our experimental results on
Github, making them available as competitive baselines. We welcome
contributions from other research groups and practitioners.
