  Video-based action recognition has recently attracted much attention in the
field of computer vision. To solve more complex recognition tasks, it has
become necessary to distinguish different levels of interclass variations.
Inspired by a common flowchart based on the human decision-making process that
first narrows down the probable classes and then applies a "rethinking" process
for finer-level recognition, we propose an effective universal-to-specific
(U2S) framework for complex action recognition. The U2S framework is composed
of three subnetworks: a universal network, a category-specific network, and a
mask network. The universal network first learns universal feature
representations. The mask network then generates attention masks for confusing
classes through category regularization based on the output of the universal
network. The mask is further used to guide the category-specific network for
class-specific feature representations. The entire framework is optimized in an
end-to-end manner. Experiments on a variety of benchmark datasets, e.g., the
Something-Something, UCF101, and HMDB51 datasets, demonstrate the effectiveness
of the U2S framework; i.e., U2S can focus on discriminative spatiotemporal
regions for confusing categories. We further visualize the relationship between
different classes, showing that U2S indeed improves the discriminability of
learned features. Moreover, the proposed U2S model is a general framework and
may adopt any base recognition network.
