  Recent efforts in multi-domain learning for semantic segmentation attempt to
learn multiple geographical datasets in a universal, joint model. A simple
fine-tuning experiment performed sequentially on three popular road scene
segmentation datasets demonstrates that existing segmentation frameworks fail
at incrementally learning on a series of visually disparate geographical
domains. When learning a new domain, the model catastrophically forgets
previously learned knowledge. In this work, we pose the problem of multi-domain
incremental learning for semantic segmentation. Given a model trained on a
particular geographical domain, the goal is to (i) incrementally learn a new
geographical domain, (ii) while retaining performance on the old domain, (iii)
given that the previous domain's dataset is not accessible. We propose a
dynamic architecture that assigns universally shared, domain-invariant
parameters to capture homogeneous semantic features present in all domains,
while dedicated domain-specific parameters learn the statistics of each domain.
Our novel optimization strategy helps achieve a good balance between retention
of old knowledge (stability) and acquiring new knowledge (plasticity). We
demonstrate the effectiveness of our proposed solution on domain incremental
settings pertaining to real-world driving scenes from roads of Germany
(Cityscapes), the United States (BDD100k), and India (IDD).
