  We explore the interpretation of sound for robot decision making, inspired by
human speech comprehension. While previous methods separate sound processing
unit and robot controller, we propose an end-to-end deep neural network which
directly interprets sound commands for visual-based decision making. The
network is trained using reinforcement learning with auxiliary losses on the
sight and sound networks. We demonstrate our approach on two robots, a
TurtleBot3 and a Kuka-IIWA arm, which hear a command word, identify the
associated target object, and perform precise control to reach the target. For
both robots, we show the effectiveness of our network in generalization to
sound types and robotic tasks empirically. We successfully transfer the policy
learned in simulator to a real-world TurtleBot3.
