  We focus on the task of unknown object rearrangement, where a robot is
supposed to re-configure the objects into a desired goal configuration
specified by an RGB-D image. Recent works explore unknown object rearrangement
systems by incorporating learning-based perception modules. However, they are
sensitive to perception error, and pay less attention to task-level
performance. In this paper, we aim to develop an effective system for unknown
object rearrangement amidst perception noise. We theoretically reveal the noisy
perception impacts grasp and place in a decoupled way, and show such a
decoupled structure is valuable to improve task optimality. We propose GSP, a
dual-loop system with the decoupled structure as prior. For the inner loop, we
learn a see policy for self-confident in-hand object matching. For the outer
loop, we learn a grasp policy aware of object matching and grasp capability
guided by task-level rewards. We leverage the foundation model CLIP for object
matching, policy learning and self-termination. A series of experiments
indicate that GSP can conduct unknown object rearrangement with higher
completion rates and fewer steps.
