  We address weakly supervised action alignment and segmentation in videos,
where only the order of occurring actions is available during training. We
propose Discriminative Differentiable Dynamic Time Warping (D3TW), the first
discriminative model using weak ordering supervision. The key technical
challenge for discriminative modeling with weak supervision is that the loss
function of the ordering supervision is usually formulated using dynamic
programming and is thus not differentiable. We address this challenge with a
continuous relaxation of the min-operator in dynamic programming and extend the
alignment loss to be differentiable. The proposed D3TW innovatively solves
sequence alignment with discriminative modeling and end-to-end training, which
substantially improves the performance in weakly supervised action alignment
and segmentation tasks. We show that our model is able to bypass the
degenerated sequence problem usually encountered in previous work and
outperform the current state-of-the-art across three evaluation metrics in two
challenging datasets.
