  Defining reward functions for skill learning has been a long-standing
challenge in robotics. Recently, vision-language models (VLMs) have shown
promise in defining reward signals for teaching robots manipulation skills.
However, existing works often provide reward guidance that is too coarse,
leading to inefficient learning processes. In this paper, we address this issue
by implementing more fine-grained reward guidance. We decompose tasks into
simpler sub-tasks, using this decomposition to offer more informative reward
guidance with VLMs. We also propose a VLM-based self imitation learning process
to speed up learning. Empirical evidence demonstrates that our algorithm
consistently outperforms baselines such as CLIP, LIV, and RoboCLIP.
Specifically, our algorithm achieves a $5.4 \times$ higher average success rate
compared to the best baseline, RoboCLIP, across a series of manipulation tasks.
