  Tool use is an important milestone in the evolution of intelligence. In this
paper, we investigate different modes of tool use that emerge in a reaching and
dragging task. In this task, a jointed arm with a gripper must grab a tool (T,
I, or L-shaped) and drag an object down to the target location (the bottom of
the arena). The simulated environment had real physics such as gravity and
friction. We trained a deep-reinforcement learning based controller (with raw
visual and proprioceptive input) with minimal reward shaping information to
tackle this task. We observed the emergence of a wide range of unexpected
behaviors, not directly encoded in the motor primitives or reward functions.
Examples include hitting the object to the target location, correcting error of
initial contact, throwing the tool toward the object, as well as normal
expected behavior such as wide sweep. Also, we further analyzed these behaviors
based on the type of tool and the initial position of the target object. Our
results show a rich repertoire of behaviors, beyond the basic built-in
mechanisms of the deep reinforcement learning method we used.
