  Comparing data defined over space and time is notoriously hard, because it
involves quantifying both spatial and temporal variability, while at the same
time taking into account the chronological structure of data. Dynamic Time
Warping (DTW) computes an optimal alignment between time series in agreement
with the chronological order, but is inherently blind to spatial shifts. In
this paper, we propose Spatio-Temporal Alignments (STA), a new differentiable
formulation of DTW, in which spatial differences between time samples are
accounted for using regularized optimal transport (OT). Our temporal alignments
are handled through a smooth variant of DTW called soft-DTW, for which we prove
a new property: soft-DTW increases quadratically with time shifts. The cost
matrix within soft-DTW that we use are computed using unbalanced OT, to handle
the case in which observations are not normalized probabilities. Experiments on
handwritten letters and brain imaging data confirm our theoretical findings and
illustrate the effectiveness of STA as a dissimilarity for spatio-temporal
data.
