  Many tasks in video analysis and understanding boil down to the need for
frame-based feature learning, aiming to encapsulate the relevant visual content
so as to enable simpler and easier subsequent processing. While supervised
strategies for this learning task can be envisioned, self and weakly-supervised
alternatives are preferred due to the difficulties in getting labeled data.
This paper introduces LRProp -- a novel weakly-supervised representation
learning approach, with an emphasis on the application of temporal alignment
between pairs of videos of the same action category. The proposed approach uses
a transformer encoder for extracting frame-level features, and employs the DTW
algorithm within the training iterations in order to identify the alignment
path between video pairs. Through a process referred to as ``pair-wise position
propagation'', the probability distributions of these correspondences per
location are matched with the similarity of the frame-level features via
KL-divergence minimization. The proposed algorithm uses also a regularized
SoftDTW loss for better tuning the learned features. Our novel representation
learning paradigm consistently outperforms the state of the art on temporal
alignment tasks, establishing a new performance bar over several downstream
video analysis applications.
