  We study the issue of catastrophic forgetting in the context of neural
multimodal approaches to Visual Question Answering (VQA). Motivated by evidence
from psycholinguistics, we devise a set of linguistically-informed VQA tasks,
which differ by the types of questions involved (Wh-questions and polar
questions). We test what impact task difficulty has on continual learning, and
whether the order in which a child acquires question types facilitates
computational models. Our results show that dramatic forgetting is at play and
that task difficulty and order matter. Two well-known current continual
learning methods mitigate the problem only to a limiting degree.
