  This paper addresses the problem of spatiotemporal localization of actions in
videos. Compared to leading approaches, which all learn to localize based on
carefully annotated boxes on training video frames, we adhere to a
weakly-supervised solution that only requires a video class label. We introduce
an actor-supervised architecture that exploits the inherent compositionality of
actions in terms of actor transformations, to localize actions. We make two
contributions. First, we propose actor proposals derived from a detector for
human and non-human actors intended for images, which is linked over time by
Siamese similarity matching to account for actor deformations. Second, we
propose an actor-based attention mechanism that enables the localization of the
actions from action class labels and actor proposals and is end-to-end
trainable. Experiments on three human and non-human action datasets show actor
supervision is state-of-the-art for weakly-supervised action localization and
is even competitive to some fully-supervised alternatives.
