  Autonomous self-improving robots that interact and improve with experience
are key to the real-world deployment of robotic systems. In this paper, we
propose an online learning method, SELFI, that leverages online robot
experience to rapidly fine-tune pre-trained control policies efficiently. SELFI
applies online model-free reinforcement learning on top of offline model-based
learning to bring out the best parts of both learning paradigms. Specifically,
SELFI stabilizes the online learning process by incorporating the same
model-based learning objective from offline pre-training into the Q-values
learned with online model-free reinforcement learning. We evaluate SELFI in
multiple real-world environments and report improvements in terms of collision
avoidance, as well as more socially compliant behavior, measured by a human
user study. SELFI enables us to quickly learn useful robotic behaviors with
less human interventions such as pre-emptive behavior for the pedestrians,
collision avoidance for small and transparent objects, and avoiding travel on
uneven floor surfaces. We provide supplementary videos to demonstrate the
performance of our fine-tuned policy on our project page.
