  Longitudinal reproducibility is an essential concern in automated medical
image segmentation, yet has proven to be an elusive objective as manual brain
structure tracings have shown more than 10% variability. To improve
reproducibility, lon-gitudinal segmentation (4D) approaches have been
investigated to reconcile tem-poral variations with traditional 3D approaches.
In the past decade, multi-atlas la-bel fusion has become a state-of-the-art
segmentation technique for 3D image and many efforts have been made to adapt it
to a 4D longitudinal fashion. However, the previous methods were either limited
by using application specified energy function (e.g., surface fusion and multi
model fusion) or only considered tem-poral smoothness on two consecutive time
points (t and t+1) under sparsity as-sumption. Therefore, a 4D multi-atlas
label fusion theory for general label fusion purpose and simultaneously
considering temporal consistency on all time points is appealing. Herein, we
propose a novel longitudinal label fusion algorithm, called 4D joint label
fusion (4DJLF), to incorporate the temporal consistency modeling via non-local
patch-intensity covariance models. The advantages of 4DJLF include: (1) 4DJLF
is under the general label fusion framework by simul-taneously incorporating
the spatial and temporal covariance on all longitudinal time points. (2) The
proposed algorithm is a longitudinal generalization of a lead-ing joint label
fusion method (JLF) that has proven adaptable to a wide variety of
applications. (3) The spatial temporal consistency of atlases is modeled in a
prob-abilistic model inspired from both voting based and statistical fusion.
The pro-posed approach improves the consistency of the longitudinal
segmentation while retaining sensitivity compared with original JLF approach
using the same set of atlases. The method is available online in open-source.
