  Predicting an interaction before it is fully executed is very important in
applications such as human-robot interaction and video surveillance. In a
two-human interaction scenario, there often contextual dependency structure
between the global interaction context of the two humans and the local context
of the different body parts of each human. In this paper, we propose to learn
the structure of the interaction contexts, and combine it with the spatial and
temporal information of a video sequence for a better prediction of the
interaction class. The structural models, including the spatial and the
temporal models, are learned with Long Short Term Memory (LSTM) networks to
capture the dependency of the global and local contexts of each RGB frame and
each optical flow image, respectively. LSTM networks are also capable of
detecting the key information from the global and local interaction contexts.
Moreover, to effectively combine the structural models with the spatial and
temporal models for interaction prediction, a ranking score fusion method is
also introduced to automatically compute the optimal weight of each model for
score fusion. Experimental results on the BIT Interaction and the
UT-Interaction datasets clearly demonstrate the benefits of the proposed
method.
