  A desirable trait of an artificial agent acting in the visual world is to
continually learn a sequence of language-informed tasks while striking a
balance between sufficiently specializing in each task and building a
generalized knowledge for transfer. Selective specialization, i.e., a careful
selection of model components to specialize in each task, is a strategy to
provide control over this trade-off. However, the design of selection
strategies requires insights on the role of each model component in learning
rather specialized or generalizable representations, which poses a gap in
current research. Thus, our aim with this work is to provide an extensive
analysis of selection strategies for visually grounded continual language
learning. Due to the lack of suitable benchmarks for this purpose, we introduce
two novel diagnostic datasets that provide enough control and flexibility for a
thorough model analysis. We assess various heuristics for module specialization
strategies as well as quantifiable measures for two different types of model
architectures. Finally, we design conceptually simple approaches based on our
analysis that outperform common continual learning baselines. Our results
demonstrate the need for further efforts towards better aligning continual
learning algorithms with the learning behaviors of individual model parts.
