  We study real-time collaborative robot (cobot) handling, where the cobot
maneuvers a workpiece under human commands. This is useful when it is risky for
humans to directly handle the workpiece. However, it is hard to make the cobot
both easy to command and flexible in possible operations. In this work, we
propose a Real-Time Collaborative Robot Handling (RTCoHand) framework that
allows the control of cobot via user-customized dynamic gestures. This is hard
due to variations among users, human motion uncertainties, and noisy human
input. We model the task as a probabilistic generative process, referred to as
Conditional Collaborative Handling Process (CCHP), and learn from human-human
collaboration. We thoroughly evaluate the adaptability and robustness of CCHP
and apply our approach to a real-time cobot handling task with Kinova Gen3
robot arm. We achieve seamless human-robot collaboration with both experienced
and new users. Compared to classical controllers, RTCoHand allows significantly
more complex maneuvers and lower user cognitive burden. It also eliminates the
need for trial-and-error, rendering it advantageous in safety-critical tasks.
