  Hands are the central means by which humans manipulate their world and being
able to reliably extract hand state information from Internet videos of humans
engaged in their hands has the potential to pave the way to systems that can
learn from petabytes of video data. This paper proposes steps towards this by
inferring a rich representation of hands engaged in interaction method that
includes: hand location, side, contact state, and a box around the object in
contact. To support this effort, we gather a large-scale dataset of hands in
contact with objects consisting of 131 days of footage as well as a 100K
annotated hand-contact video frame dataset. The learned model on this dataset
can serve as a foundation for hand-contact understanding in videos. We
quantitatively evaluate it both on its own and in service of predicting and
learning from 3D meshes of human hands.
