  Compositional Natural Language Inference has been explored to assess the true
abilities of neural models to perform NLI. Yet, current evaluations assume
models to have full access to all primitive inferences in advance, in contrast
to humans that continuously acquire inference knowledge. In this paper, we
introduce the Continual Compositional Generalization in Inference (C2Gen NLI)
challenge, where a model continuously acquires knowledge of constituting
primitive inference tasks as a basis for compositional inferences. We explore
how continual learning affects compositional generalization in NLI, by
designing a continual learning setup for compositional NLI inference tasks. Our
experiments demonstrate that models fail to compositionally generalize in a
continual scenario. To address this problem, we first benchmark various
continual learning algorithms and verify their efficacy. We then further
analyze C2Gen, focusing on how to order primitives and compositional inference
types and examining correlations between subtasks. Our analyses show that by
learning subtasks continuously while observing their dependencies and
increasing degrees of difficulty, continual learning can enhance composition
generalization ability.
