  This paper proposes a learning-based visual peg-in-hole that enables training
with several shapes in simulation, and adapting to arbitrary unseen shapes in
real world with minimal sim-to-real cost. The core idea is to decouple the
generalization of the sensory-motor policy to the design of a fast-adaptable
perception module and a simulated generic policy module. The framework consists
of a segmentation network (SN), a virtual sensor network (VSN), and a
controller network (CN). Concretely, the VSN is trained to measure the pose of
the unseen shape from a segmented image. After that, given the shape-agnostic
pose measurement, the CN is trained to achieve generic peg-in-hole. Finally,
when applying to real unseen holes, we only have to fine-tune the SN required
by the simulated VSN+CN. To further minimize the transfer cost, we propose to
automatically collect and annotate the data for the SN after one-minute human
teaching. Simulated and real-world results are presented under the
configurations of eye-to/in-hand. An electric vehicle charging system with the
proposed policy inside achieves a 10/10 success rate in 2-3s, using only
hundreds of auto-labeled samples for the SN transfer.
