  Developing end-to-end action recognition models on long videos is fundamental
and crucial for long-video action understanding. Due to the unaffordable cost
of end-to-end training on the whole long videos, existing works generally train
models on short clips trimmed from long videos. However, this
``trimming-then-training'' practice requires action interval annotations for
clip-level supervision, i.e., knowing which actions are trimmed into the clips.
Unfortunately, collecting such annotations is very expensive and prevents model
training at scale. To this end, this work aims to build a weakly supervised
end-to-end framework for training recognition models on long videos, with only
video-level action category labels. Without knowing the precise temporal
locations of actions in long videos, our proposed weakly supervised framework,
namely AdaptFocus, estimates where and how likely the actions will occur to
adaptively focus on informative action clips for end-to-end training. The
effectiveness of the proposed AdaptFocus framework is demonstrated on three
long-video datasets. Furthermore, for downstream long-video tasks, our
AdaptFocus framework provides a weakly supervised feature extraction pipeline
for extracting more robust long-video features, such that the state-of-the-art
methods on downstream tasks are significantly advanced. We will release the
code and models.
