  Behavior cloning (BC) currently stands as a dominant paradigm for learning
real-world visual manipulation. However, in tasks that require locally
corrective behaviors like multi-part assembly, learning robust policies purely
from human demonstrations remains challenging. Reinforcement learning (RL) can
mitigate these limitations by allowing policies to acquire locally corrective
behaviors through task reward supervision and exploration. This paper explores
the use of RL fine-tuning to improve upon BC-trained policies in precise
manipulation tasks. We analyze and overcome technical challenges associated
with using RL to directly train policy networks that incorporate modern
architectural components like diffusion models and action chunking. We propose
training residual policies on top of frozen BC-trained diffusion models using
standard policy gradient methods and sparse rewards, an approach we call ResiP
(Residual for Precise manipulation). Our experimental results demonstrate that
this residual learning framework can significantly improve success rates beyond
the base BC-trained models in high-precision assembly tasks by learning
corrective actions. We also show that by combining ResiP with teacher-student
distillation and visual domain randomization, our method can enable learning
real-world policies for robotic assembly directly from RGB images. Find videos
and code at \url{https://residual-assembly.github.io}.
