  Policies learned through Reinforcement Learning (RL) and Imitation Learning
(IL) have demonstrated significant potential in achieving advanced performance
in continuous control tasks. However, in real-world environments, it is often
necessary to further customize a trained policy when there are additional
requirements that were unforeseen during the original training phase. It is
possible to fine-tune the policy to meet the new requirements, but this often
requires collecting new data with the added requirements and access to the
original training metric and policy parameters. In contrast, an online planning
algorithm, if capable of meeting the additional requirements, can eliminate the
necessity for extensive training phases and customize the policy without
knowledge of the original training scheme or task. In this work, we propose a
generic online planning algorithm for customizing continuous-control policies
at the execution time which we call Residual-MPPI. It is able to customize a
given prior policy on new performance metrics in few-shot and even zero-shot
online settings. Also, Residual-MPPI only requires access to the action
distribution produced by the prior policy, without additional knowledge
regarding the original task. Through our experiments, we demonstrate that the
proposed Residual-MPPI algorithm can accomplish the few-shot/zero-shot online
policy customization task effectively, including customizing the champion-level
racing agent, Gran Turismo Sophy (GT Sophy) 1.0, in the challenging car racing
scenario, Gran Turismo Sport (GTS) environment. Demo videos are available on
our website: https://sites.google.com/view/residual-mppi
