  We are given a set of video clips, each one annotated with an {\em ordered}
list of actions, such as "walk" then "sit" then "answer phone" extracted from,
for example, the associated text script. We seek to temporally localize the
individual actions in each clip as well as to learn a discriminative classifier
for each action. We formulate the problem as a weakly supervised temporal
assignment with ordering constraints. Each video clip is divided into small
time intervals and each time interval of each video clip is assigned one action
label, while respecting the order in which the action labels appear in the
given annotations. We show that the action label assignment can be determined
together with learning a classifier for each action in a discriminative manner.
We evaluate the proposed model on a new and challenging dataset of 937 video
clips with a total of 787720 frames containing sequences of 16 different
actions from 69 Hollywood movies.
