  Learning effective representations of visual data that generalize to a
variety of downstream tasks has been a long quest for computer vision. Most
representation learning approaches rely solely on visual data such as images or
videos. In this paper, we explore a novel approach, where we use human
interaction and attention cues to investigate whether we can learn better
representations compared to visual-only representations. For this study, we
collect a dataset of human interactions capturing body part movements and gaze
in their daily lives. Our experiments show that our "muscly-supervised"
representation that encodes interaction and attention cues outperforms a
visual-only state-of-the-art method MoCo (He et al.,2020), on a variety of
target tasks: scene classification (semantic), action recognition (temporal),
depth estimation (geometric), dynamics prediction (physics) and walkable
surface estimation (affordance). Our code and dataset are available at:
https://github.com/ehsanik/muscleTorch.
