  We strive for spatio-temporal localization of actions in videos. The
state-of-the-art relies on action proposals at test time and selects the best
one with a classifier trained on carefully annotated box annotations.
Annotating action boxes in video is cumbersome, tedious, and error prone.
Rather than annotating boxes, we propose to annotate actions in video with
points on a sparse subset of frames only. We introduce an overlap measure
between action proposals and points and incorporate them all into the objective
of a non-convex Multiple Instance Learning optimization. Experimental
evaluation on the UCF Sports and UCF 101 datasets shows that (i)
spatio-temporal proposals can be used to train classifiers while retaining the
localization performance, (ii) point annotations yield results comparable to
box annotations while being significantly faster to annotate, (iii) with a
minimum amount of supervision our approach is competitive to the
state-of-the-art. Finally, we introduce spatio-temporal action annotations on
the train and test videos of Hollywood2, resulting in Hollywood2Tubes,
available at http://tinyurl.com/hollywood2tubes.
