  We propose to leverage a real-world, human activity RGB dataset to teach a
robot Task-Oriented Grasping (TOG). We develop a model that takes as input an
RGB image and outputs a hand pose and configuration as well as an object pose
and a shape. We follow the insight that jointly estimating hand and object
poses increases accuracy compared to estimating these quantities independently
of each other. Given the trained model, we process an RGB dataset to
automatically obtain the data to train a TOG model. This model takes as input
an object point cloud and outputs a suitable region for task-specific grasping.
Our ablation study shows that training an object pose predictor with the hand
pose information (and vice versa) is better than training without this
information. Furthermore, our results on a real-world dataset show the
applicability and competitiveness of our method over state-of-the-art.
Experiments with a robot demonstrate that our method can allow a robot to
preform TOG on novel objects.
