  Explainability methods for NLP systems encounter a version of the fundamental
problem of causal inference: for a given ground-truth input text, we never
truly observe the counterfactual texts necessary for isolating the causal
effects of model representations on outputs. In response, many explainability
methods make no use of counterfactual texts, assuming they will be unavailable.
In this paper, we show that robust causal explainability methods can be created
using approximate counterfactuals, which can be written by humans to
approximate a specific counterfactual or simply sampled using metadata-guided
heuristics. The core of our proposal is the Causal Proxy Model (CPM). A CPM
explains a black-box model $\mathcal{N}$ because it is trained to have the same
actual input/output behavior as $\mathcal{N}$ while creating neural
representations that can be intervened upon to simulate the counterfactual
input/output behavior of $\mathcal{N}$. Furthermore, we show that the best CPM
for $\mathcal{N}$ performs comparably to $\mathcal{N}$ in making factual
predictions, which means that the CPM can simply replace $\mathcal{N}$, leading
to more explainable deployed models. Our code is available at
https://github.com/frankaging/Causal-Proxy-Model.
