  Visual Question Answering (VQA) is a multi-discipline research task. To
produce the right answer, it requires an understanding of the visual content of
images, the natural language questions, as well as commonsense reasoning over
the information contained in the image and world knowledge. Recently,
large-scale Vision-and-Language Pre-trained Models (VLPMs) have been the
mainstream approach to VQA tasks due to their superior performance. The
standard practice is to fine-tune large-scale VLPMs pre-trained on huge
general-domain datasets using the domain-specific VQA datasets. However, in
reality, the application domain can change over time, necessitating VLPMs to
continually learn and adapt to new domains without forgetting previously
acquired knowledge. Most existing continual learning (CL) research concentrates
on unimodal tasks, whereas a more practical application scenario, i.e, CL on
cross-domain VQA, has not been studied. Motivated by this, we introduce
CL-CrossVQA, a rigorous Continual Learning benchmark for Cross-domain Visual
Question Answering, through which we conduct extensive experiments on 4 VLPMs,
4 CL approaches, and 5 VQA datasets from different domains. In addition, by
probing the forgetting phenomenon of the intermediate layers, we provide
insights into how model architecture affects CL performance, why CL approaches
can help mitigate forgetting in VLPMs to some extent, and how to design CL
approaches suitable for VLPMs in this challenging continual learning
environment. To facilitate future work on CL for cross-domain VQA, we will
release our datasets and code.
