  We present an end-to-end deep Convolutional Neural Network called
Convolutional Relational Machine (CRM) for recognizing group activities that
utilizes the information in spatial relations between individual persons in
image or video. It learns to produce an intermediate spatial representation
(activity map) based on individual and group activities. A multi-stage
refinement component is responsible for decreasing the incorrect predictions in
the activity map. Finally, an aggregation component uses the refined
information to recognize group activities. Experimental results demonstrate the
constructive contribution of the information extracted and represented in the
form of the activity map. CRM shows advantages over state-of-the-art models on
Volleyball and Collective Activity datasets.
