  Multiple sclerosis lesion activity segmentation is the task of detecting new
and enlarging lesions that appeared between a baseline and a follow-up brain
MRI scan. While deep learning methods for single-scan lesion segmentation are
common, deep learning approaches for lesion activity have only been proposed
recently. Here, a two-path architecture processes two 3D MRI volumes from two
time points. In this work, we investigate whether extending this problem to
full 4D deep learning using a history of MRI volumes and thus an extended
baseline can improve performance. For this purpose, we design a recurrent
multi-encoder-decoder architecture for processing 4D data. We find that adding
more temporal information is beneficial and our proposed architecture
outperforms previous approaches with a lesion-wise true positive rate of 0.84
at a lesion-wise false positive rate of 0.19.
