  This paper introduces a unified framework for video action segmentation via
sequence to sequence (seq2seq) translation in a fully and timestamp supervised
setup. In contrast to current state-of-the-art frame-level prediction methods,
we view action segmentation as a seq2seq translation task, i.e., mapping a
sequence of video frames to a sequence of action segments. Our proposed method
involves a series of modifications and auxiliary loss functions on the standard
Transformer seq2seq translation model to cope with long input sequences opposed
to short output sequences and relatively few videos. We incorporate an
auxiliary supervision signal for the encoder via a frame-wise loss and propose
a separate alignment decoder for an implicit duration prediction. Finally, we
extend our framework to the timestamp supervised setting via our proposed
constrained k-medoids algorithm to generate pseudo-segmentations. Our proposed
framework performs consistently on both fully and timestamp supervised
settings, outperforming or competing state-of-the-art on several datasets. Our
code is publicly available at https://github.com/boschresearch/UVAST.
