  Selective rationalization explains the prediction of complex neural networks
by finding a small subset of the input that is sufficient to predict the neural
model output. The selection mechanism is commonly integrated into the model
itself by specifying a two-component cascaded system consisting of a rationale
generator, which makes a binary selection of the input features (which is the
rationale), and a predictor, which predicts the output based only on the
selected features. The components are trained jointly to optimize prediction
performance. In this paper, we reveal a major problem with such cooperative
rationalization paradigm -- model interlocking. Interlocking arises when the
predictor overfits to the features selected by the generator thus reinforcing
the generator's selection even if the selected rationales are sub-optimal. The
fundamental cause of the interlocking problem is that the rationalization
objective to be minimized is concave with respect to the generator's selection
policy. We propose a new rationalization framework, called A2R, which
introduces a third component into the architecture, a predictor driven by soft
attention as opposed to selection. The generator now realizes both soft and
hard attention over the features and these are fed into the two different
predictors. While the generator still seeks to support the original predictor
performance, it also minimizes a gap between the two predictors. As we will
show theoretically, since the attention-based predictor exhibits a better
convexity property, A2R can overcome the concavity barrier. Our experiments on
two synthetic benchmarks and two real datasets demonstrate that A2R can
significantly alleviate the interlock problem and find explanations that better
align with human judgments. We release our code at
https://github.com/Gorov/Understanding_Interlocking.
