  Decisions of complex language understanding models can be rationalized by
limiting their inputs to a relevant subsequence of the original text. A
rationale should be as concise as possible without significantly degrading task
performance, but this balance can be difficult to achieve in practice. In this
paper, we show that it is possible to better manage this trade-off by
optimizing a bound on the Information Bottleneck (IB) objective. Our fully
unsupervised approach jointly learns an explainer that predicts sparse binary
masks over sentences, and an end-task predictor that considers only the
extracted rationale. Using IB, we derive a learning objective that allows
direct control of mask sparsity levels through a tunable sparse prior.
Experiments on ERASER benchmark tasks demonstrate significant gains over
norm-minimization techniques for both task performance and agreement with human
rationales. Furthermore, we find that in the semi-supervised setting, a modest
amount of gold rationales (25% of training examples) closes the gap with a
model that uses the full input.
