  We address the problem of detecting attention targets in video. Our goal is
to identify where each person in each frame of a video is looking, and
correctly handle the case where the gaze target is out-of-frame. Our novel
architecture models the dynamic interaction between the scene and head features
and infers time-varying attention targets. We introduce a new annotated
dataset, VideoAttentionTarget, containing complex and dynamic patterns of
real-world gaze behavior. Our experiments show that our model can effectively
infer dynamic attention in videos. In addition, we apply our predicted
attention maps to two social gaze behavior recognition tasks, and show that the
resulting classifiers significantly outperform existing methods. We achieve
state-of-the-art performance on three datasets: GazeFollow (static images),
VideoAttentionTarget (videos), and VideoCoAtt (videos), and obtain the first
results for automatically classifying clinically-relevant gaze behavior without
wearable cameras or eye trackers.
