  Action detection plays an important role in high-level video understanding
and media interpretation. Many existing studies fulfill this spatio-temporal
localization by modeling the context, capturing the relationship of actors,
objects, and scenes conveyed in the video. However, they often universally
treat all the actors without considering the consistency and distinctness
between individuals, leaving much room for improvement. In this paper, we
explicitly highlight the identity information of the actors in terms of both
long-term and short-term context through a graph memory network, namely
identity-aware graph memory network (IGMN). Specifically, we propose the
hierarchical graph neural network (HGNN) to comprehensively conduct long-term
relation modeling within the same identity as well as between different ones.
Regarding short-term context, we develop a dual attention module (DAM) to
generate identity-aware constraint to reduce the influence of interference by
the actors of different identities. Extensive experiments on the challenging
AVA dataset demonstrate the effectiveness of our method, which achieves
state-of-the-art results on AVA v2.1 and v2.2.
