  We revisit the study of a wrist-mounted camera system (referred to as
HandCam) for recognizing activities of hands. HandCam has two unique properties
as compared to egocentric systems (referred to as HeadCam): (1) it avoids the
need to detect hands; (2) it more consistently observes the activities of
hands. By taking advantage of these properties, we propose a
deep-learning-based method to recognize hand states (free v.s. active hands,
hand gestures, object categories), and discover object categories. Moreover, we
propose a novel two-streams deep network to further take advantage of both
HandCam and HeadCam. We have collected a new synchronized HandCam and HeadCam
dataset with 20 videos captured in three scenes for hand states recognition.
Experiments show that our HandCam system consistently outperforms a
deep-learning-based HeadCam method (with estimated manipulation regions) and a
dense-trajectory-based HeadCam method in all tasks. We also show that HandCam
videos captured by different users can be easily aligned to improve free v.s.
active recognition accuracy (3.3% improvement) in across-scenes use case.
Moreover, we observe that finetuning Convolutional Neural Network consistently
improves accuracy. Finally, our novel two-streams deep network combining
HandCam and HeadCam features achieves the best performance in four out of five
tasks. With more data, we believe a joint HandCam and HeadCam system can
robustly log hand states in daily life.
