  Recently, large-scale pre-trained Vision-and-Language (VL) foundation models
have demonstrated remarkable capabilities in many zero-shot downstream tasks,
achieving competitive results for recognizing objects defined by as little as
short text prompts. However, it has also been shown that VL models are still
brittle in Structured VL Concept (SVLC) reasoning, such as the ability to
recognize object attributes, states, and inter-object relations. This leads to
reasoning mistakes, which need to be corrected as they occur by teaching VL
models the missing SVLC skills; often this must be done using private data
where the issue was found, which naturally leads to a data-free continual (no
task-id) VL learning setting. In this work, we introduce the first Continual
Data-Free Structured VL Concepts Learning (ConStruct-VL) benchmark and show it
is challenging for many existing data-free CL strategies. We, therefore,
propose a data-free method comprised of a new approach of Adversarial
Pseudo-Replay (APR) which generates adversarial reminders of past tasks from
past task models. To use this method efficiently, we also propose a continual
parameter-efficient Layered-LoRA (LaLo) neural architecture allowing
no-memory-cost access to all past models at train time. We show this approach
outperforms all data-free methods by as much as ~7% while even matching some
levels of experience-replay (prohibitive for applications where data-privacy
must be preserved). Our code is publicly available at
https://github.com/jamessealesmith/ConStruct-VL
