  Convolutional neural networks (CNNs) have shown promising results on several
segmentation tasks in magnetic resonance (MR) images. However, the accuracy of
CNNs may degrade severely when segmenting images acquired with different
scanners and/or protocols as compared to the training data, thus limiting their
practical utility. We address this shortcoming in a lifelong multi-domain
learning setting by treating images acquired with different scanners or
protocols as samples from different, but related domains. Our solution is a
single CNN with shared convolutional filters and domain-specific batch
normalization layers, which can be tuned to new domains with only a few
($\approx$ 4) labelled images. Importantly, this is achieved while retaining
performance on the older domains whose training data may no longer be
available. We evaluate the method for brain structure segmentation in MR
images. Results demonstrate that the proposed method largely closes the gap to
the benchmark, which is training a dedicated CNN for each scanner.
