  Modeling relation between actors is important for recognizing group activity
in a multi-person scene. This paper aims at learning discriminative relation
between actors efficiently using deep models. To this end, we propose to build
a flexible and efficient Actor Relation Graph (ARG) to simultaneously capture
the appearance and position relation between actors. Thanks to the Graph
Convolutional Network, the connections in ARG could be automatically learned
from group activity videos in an end-to-end manner, and the inference on ARG
could be efficiently performed with standard matrix operations. Furthermore, in
practice, we come up with two variants to sparsify ARG for more effective
modeling in videos: spatially localized ARG and temporal randomized ARG. We
perform extensive experiments on two standard group activity recognition
datasets: the Volleyball dataset and the Collective Activity dataset, where
state-of-the-art performance is achieved on both datasets. We also visualize
the learned actor graphs and relation features, which demonstrate that the
proposed ARG is able to capture the discriminative relation information for
group activity recognition.
