  In clinical settings, where acquisition conditions and patient populations
change over time, continual learning is key for ensuring the safe use of deep
neural networks. Yet most existing work focuses on convolutional architectures
and image classification. Instead, radiologists prefer to work with
segmentation models that outline specific regions-of-interest, for which
Transformer-based architectures are gaining traction. The self-attention
mechanism of Transformers could potentially mitigate catastrophic forgetting,
opening the way for more robust medical image segmentation. In this work, we
explore how recently-proposed Transformer mechanisms for semantic segmentation
behave in sequential learning scenarios, and analyse how best to adapt
continual learning strategies for this setting. Our evaluation on hippocampus
segmentation shows that Transformer mechanisms mitigate catastrophic forgetting
for medical image segmentation compared to purely convolutional architectures,
and demonstrates that regularising ViT modules should be done with caution.
