  Robotic peg-in-hole assembly remains a challenging task due to its high
accuracy demand. Previous work tends to simplify the problem by restricting the
degree of freedom of the end-effector, or limiting the distance between the
target and the initial pose position, which prevents them from being deployed
in real-world manufacturing. Thus, we present a Coarse-to-Fine Visual Servoing
(CFVS) peg-in-hole method, achieving 6-DoF end-effector motion control based on
3D visual feedback. CFVS can handle arbitrary tilt angles and large initial
alignment errors through a fast pose estimation before refinement. Furthermore,
by introducing a confidence map to ignore the irrelevant contour of objects,
CFVS is robust against noise and can deal with various targets beyond training
data. Extensive experiments show CFVS outperforms state-of-the-art methods and
obtains 100%, 91%, and 82% average success rates in 3-DoF, 4-DoF, and 6-DoF
peg-in-hole, respectively.
