  This paper presents a teleoperation system that includes robot perception and
intent prediction from hand gestures. The perception module identifies the
objects present in the robot workspace and the intent prediction module which
object the user likely wants to grasp. This architecture allows the approach to
rely on traded control instead of direct control: we use hand gestures to
specify the goal objects for a sequential manipulation task, the robot then
autonomously generates a grasping or a retrieving motion using trajectory
optimization. The perception module relies on the model-based tracker to
precisely track the 6D pose of the objects and makes use of a state of the art
learning-based object detection and segmentation method, to initialize the
tracker by automatically detecting objects in the scene. Goal objects are
identified from user hand gestures using a trained a multi-layer perceptron
classifier. After presenting all the components of the system and their
empirical evaluation, we present experimental results comparing our pipeline to
a direct traded control approach (i.e., one that does not use prediction) which
shows that using intent prediction allows to bring down the overall task
execution time.
