  Phenomenon-specific "adversarial" datasets have been recently designed to
perform targeted stress-tests for particular inference types. Recent work (Liu
et al., 2019a) proposed that such datasets can be utilized for training NLI and
other types of models, often allowing to learn the phenomenon in focus and
improve on the challenge dataset, indicating a "blind spot" in the original
training data. Yet, although a model can improve in such a training process, it
might still be vulnerable to other challenge datasets targeting the same
phenomenon but drawn from a different distribution, such as having a different
syntactic complexity level. In this work, we extend this method to drive
conclusions about a model's ability to learn and generalize a target phenomenon
rather than to "learn" a dataset, by controlling additional aspects in the
adversarial datasets. We demonstrate our approach on two inference phenomena -
dative alternation and numerical reasoning, elaborating, and in some cases
contradicting, the results of Liu et al.. Our methodology enables building
better challenge datasets for creating more robust models, and may yield better
model understanding and subsequent overarching improvements.
