  The objective of this work is to enable manipulation tasks with respect to
the 6D pose of a dynamically moving object using a camera mounted on a robot.
Examples include maintaining a constant relative 6D pose of the robot arm with
respect to the object, grasping the dynamically moving object, or
co-manipulating the object together with a human. Fast and accurate 6D pose
estimation is crucial to achieve smooth and stable robot control in such
situations. The contributions of this work are three fold. First, we propose a
new visual perception module that asynchronously combines accurate
learning-based 6D object pose localizer and a high-rate model-based 6D pose
tracker. The outcome is a low-latency accurate and temporally consistent 6D
object pose estimation from the input video stream at up to 120 Hz. Second, we
develop a visually guided robot arm controller that combines the new visual
perception module with a torque-based model predictive control algorithm.
Asynchronous combination of the visual and robot proprioception signals at
their corresponding frequencies results in stable and robust 6D object pose
guided robot arm control. Third, we experimentally validate the proposed
approach on a challenging 6D pose estimation benchmark and demonstrate 6D
object pose-guided control with dynamically moving objects on a real 7 DoF
Franka Emika Panda robot.
