  Medical image distributions shift constantly due to changes in patient
population and discrepancies in image acquisition. These distribution changes
result in performance deterioration; deterioration that continual learning aims
to alleviate. However, only adaptation with data rehearsal strategies yields
practically desirable performance for medical image segmentation. Such
rehearsal violates patient privacy and, as most continual learning approaches,
overlooks unexpected changes from out-of-distribution instances. To transcend
both of these challenges, we introduce a distribution-aware replay strategy
that mitigates forgetting through auto-encoding of features, while
simultaneously leveraging the learned distribution of features to detect model
failure. We provide empirical corroboration on hippocampus and prostate MRI
segmentation.
