  Research in the field of Continual Semantic Segmentation is mainly
investigating novel learning algorithms to overcome catastrophic forgetting of
neural networks. Most recent publications have focused on improving learning
algorithms without distinguishing effects caused by the choice of neural
architecture.Therefore, we study how the choice of neural network architecture
affects catastrophic forgetting in class- and domain-incremental semantic
segmentation. Specifically, we compare the well-researched CNNs to recently
proposed Transformers and Hybrid architectures, as well as the impact of the
choice of novel normalization layers and different decoder heads. We find that
traditional CNNs like ResNet have high plasticity but low stability, while
transformer architectures are much more stable. When the inductive biases of
CNN architectures are combined with transformers in hybrid architectures, it
leads to higher plasticity and stability. The stability of these models can be
explained by their ability to learn general features that are robust against
distribution shifts. Experiments with different normalization layers show that
Continual Normalization achieves the best trade-off in terms of adaptability
and stability of the model. In the class-incremental setting, the choice of the
normalization layer has much less impact. Our experiments suggest that the
right choice of architecture can significantly reduce forgetting even with
naive fine-tuning and confirm that for real-world applications, the
architecture is an important factor in designing a continual learning model.
