  Rationales, snippets of extracted text that explain an inference, have
emerged as a popular framework for interpretable natural language processing
(NLP). Rationale models typically consist of two cooperating modules: a
selector and a classifier with the goal of maximizing the mutual information
(MMI) between the "selected" text and the document label. Despite their
promises, MMI-based methods often pick up on spurious text patterns and result
in models with nonsensical behaviors. In this work, we investigate whether
counterfactual data augmentation (CDA), without human assistance, can improve
the performance of the selector by lowering the mutual information between
spurious signals and the document label. Our counterfactuals are produced in an
unsupervised fashion using class-dependent generative models. From an
information theoretic lens, we derive properties of the unaugmented dataset for
which our CDA approach would succeed. The effectiveness of CDA is empirically
evaluated by comparing against several baselines including an improved
MMI-based rationale schema on two multi aspect datasets. Our results show that
CDA produces rationales that better capture the signal of interest.
