  Counterfactual explanations (CEs) are a powerful means for understanding how
decisions made by algorithms can be changed. Researchers have proposed a number
of desiderata that CEs should meet to be practically useful, such as requiring
minimal effort to enact, or complying with causal models. We consider a further
aspect to improve the usability of CEs: robustness to adverse perturbations,
which may naturally happen due to unfortunate circumstances. Since CEs
typically prescribe a sparse form of intervention (i.e., only a subset of the
features should be changed), we study the effect of addressing robustness
separately for the features that are recommended to be changed and those that
are not. Our definitions are workable in that they can be incorporated as
penalty terms in the loss functions that are used for discovering CEs. To
experiment with robustness, we create and release code where five data sets
(commonly used in the field of fair and explainable machine learning) have been
enriched with feature-specific annotations that can be used to sample
meaningful perturbations. Our experiments show that CEs are often not robust
and, if adverse perturbations take place (even if not worst-case), the
intervention they prescribe may require a much larger cost than anticipated, or
even become impossible. However, accounting for robustness in the search
process, which can be done rather easily, allows discovering robust CEs
systematically. Robust CEs make additional intervention to contrast
perturbations much less costly than non-robust CEs. We also find that
robustness is easier to achieve for the features to change, posing an important
point of consideration for the choice of what counterfactual explanation is
best for the user. Our code is available at:
https://github.com/marcovirgolin/robust-counterfactuals.
