  Action segmentation is the task of predicting an action label for each frame
of an untrimmed video. As obtaining annotations to train an approach for action
segmentation in a fully supervised way is expensive, various approaches have
been proposed to train action segmentation models using different forms of weak
supervision, e.g., action transcripts, action sets, or more recently
timestamps. Timestamp supervision is a promising type of weak supervision as
obtaining one timestamp per action is less expensive than annotating all
frames, but it provides more information than other forms of weak supervision.
However, previous works assume that every action instance is annotated with a
timestamp, which is a restrictive assumption since it assumes that annotators
do not miss any action. In this work, we relax this restrictive assumption and
take missing annotations for some action instances into account. We show that
our approach is more robust to missing annotations compared to other approaches
and various baselines.
