  This paper presents a novel approach, termed {\em Temporal Latent Residual
Network (TLRN)}, to predict a sequence of deformation fields in time-series
image registration. The challenge of registering time-series images often lies
in the occurrence of large motions, especially when images differ significantly
from a reference (e.g., the start of a cardiac cycle compared to the peak
stretching phase). To achieve accurate and robust registration results, we
leverage the nature of motion continuity and exploit the temporal smoothness in
consecutive image frames. Our proposed TLRN highlights a temporal residual
network with residual blocks carefully designed in latent deformation spaces,
which are parameterized by time-sequential initial velocity fields. We treat a
sequence of residual blocks over time as a dynamic training system, where each
block is designed to learn the residual function between desired deformation
features and current input accumulated from previous time frames. We validate
the effectivenss of TLRN on both synthetic data and real-world cine cardiac
magnetic resonance (CMR) image videos. Our experimental results shows that TLRN
is able to achieve substantially improved registration accuracy compared to the
state-of-the-art. Our code is publicly available at
https://github.com/nellie689/TLRN.
