  Human-Robot collaboration in home and industrial workspaces is on the rise.
However, the communication between robots and humans is a bottleneck. Although
people use a combination of different types of gestures to complement speech,
only a few robotic systems utilize gestures for communication. In this paper,
we propose a gesture pseudo-language and show how multiple types of gestures
can be combined to express human intent to a robot (i.e., expressing both the
desired action and its parameters - e.g., pointing to an object and showing
that the object should be emptied into a bowl). The demonstrated gestures and
the perceived table-top scene (object poses detected by CosyPose) are processed
in real-time) to extract the human's intent. We utilize behavior trees to
generate reactive robot behavior that handles various possible states of the
world (e.g., a drawer has to be opened before an object is placed into it) and
recovers from errors (e.g., when the scene changes). Furthermore, our system
enables switching between direct teleoperation of the end-effector and
high-level operation using the proposed gesture sentences. The system is
evaluated on increasingly complex tasks using a real 7-DoF Franka Emika Panda
manipulator. Controlling the robot via action gestures lowered the execution
time by up to 60%, compared to direct teleoperation.
