  Although counterfactual explanations are a popular approach to explain ML
black-box classifiers, they are less widespread in NLP. Most methods find those
explanations by iteratively perturbing the target document until it is
classified differently by the black box. We identify two main families of
counterfactual explanation methods in the literature, namely, (a)
\emph{transparent} methods that perturb the target by adding, removing, or
replacing words, and (b) \emph{opaque} approaches that project the target
document into a latent, non-interpretable space where the perturbation is
carried out subsequently. This article offers a comparative study of the
performance of these two families of methods on three classical NLP tasks. Our
empirical evidence shows that opaque approaches can be an overkill for
downstream applications such as fake news detection or sentiment analysis since
they add an additional level of complexity with no significant performance
gain. These observations motivate our discussion, which raises the question of
whether it makes sense to explain a black box using another black box.
