  The goal of this work is spatio-temporal action localization in videos, using
only the supervision from video-level class labels. The state-of-the-art casts
this weakly-supervised action localization regime as a Multiple Instance
Learning problem, where instances are a priori computed spatio-temporal
proposals. Rather than disconnecting the spatio-temporal learning from the
training, we propose Spatio-Temporal Instance Learning, which enables action
localization directly from box proposals in video frames. We outline the
assumptions of our model and propose a max-margin objective and optimization
with latent variables that enable spatio-temporal learning of actions from
video labels. We also provide an efficient linking algorithm and two reranking
strategies to facilitate and further improve the action localization.
Experimental evaluation on four action datasets demonstrate the effectiveness
of our approach for localization from weak supervision. Moreover, we show how
to incorporate other supervision levels and mixtures, as a step towards
determining optimal supervision strategies for action localization.
