  Group activity detection (GAD) is the task of identifying members of each
group and classifying the activity of the group at the same time in a video.
While GAD has been studied recently, there is still much room for improvement
in both dataset and methodology due to their limited capability to address
practical GAD scenarios. To resolve these issues, we first present a new
dataset, dubbed Caf\'e. Unlike existing datasets, Caf\'e is constructed
primarily for GAD and presents more practical scenarios and metrics, as well as
being large-scale and providing rich annotations. Along with the dataset, we
propose a new GAD model that deals with an unknown number of groups and latent
group members efficiently and effectively. We evaluated our model on three
datasets including Caf\'e, where it outperformed previous work in terms of both
accuracy and inference speed.
