  Skeletal Action recognition from an egocentric view is important for
applications such as interfaces in AR/VR glasses and human-robot interaction,
where the device has limited resources. Most of the existing skeletal action
recognition approaches use 3D coordinates of hand joints and 8-corner
rectangular bounding boxes of objects as inputs, but they do not capture how
the hands and objects interact with each other within the spatial context. In
this paper, we present a new framework called Contact-aware Skeletal Action
Recognition (CaSAR). It uses novel representations of hand-object interaction
that encompass spatial information: 1) contact points where the hand joints
meet the objects, 2) distant points where the hand joints are far away from the
object and nearly not involved in the current action. Our framework is able to
learn how the hands touch or stay away from the objects for each frame of the
action sequence, and use this information to predict the action class. We
demonstrate that our approach achieves the state-of-the-art accuracy of 91.3%
and 98.4% on two public datasets, H2O and FPHA, respectively.
