  Intelligent agents must be able to think fast and slow to perform elaborate
manipulation tasks. Reinforcement Learning (RL) has led to many promising
results on a range of challenging decision-making tasks. However, in real-world
robotics, these methods still struggle, as they require large amounts of
expensive interactions and have slow feedback loops. On the other hand, fast
human-like adaptive control methods can optimize complex robotic interactions,
yet fail to integrate multimodal feedback needed for unstructured tasks. In
this work, we propose to factor the learning problem in a hierarchical learning
and adaption architecture to get the best of both worlds. The framework
consists of two components, a slow reinforcement learning policy optimizing the
task strategy given multimodal observations, and a fast, real-time adaptive
control policy continuously optimizing the motion, stability, and effort of the
manipulator. We combine these components through a bio-inspired action space
that we call AFORCE. We demonstrate the new action space on a contact-rich
manipulation task on real hardware and evaluate its performance on three
simulated manipulation tasks. Our experiments show that AFORCE drastically
improves sample efficiency while reducing energy consumption and improving
safety.
