  Learning an egocentric action recognition model from video data is
challenging due to distractors (e.g., irrelevant objects) in the background.
Further integrating object information into an action model is hence
beneficial. Existing methods often leverage a generic object detector to
identify and represent the objects in the scene. However, several important
issues remain. Object class annotations of good quality for the target domain
(dataset) are still required for learning good object representation. Besides,
previous methods deeply couple the existing action models and need to retrain
them jointly with object representation, leading to costly and inflexible
integration. To overcome both limitations, we introduce Self-Supervised
Learning Over Sets (SOS), an approach to pre-train a generic Objects In Contact
(OIC) representation model from video object regions detected by an
off-the-shelf hand-object contact detector. Instead of augmenting object
regions individually as in conventional self-supervised learning, we view the
action process as a means of natural data transformations with unique
spatio-temporal continuity and exploit the inherent relationships among
per-video object sets. Extensive experiments on two datasets, EPIC-KITCHENS-100
and EGTEA, show that our OIC significantly boosts the performance of multiple
state-of-the-art video classification models.
