  This paper primarily focuses on evaluating and benchmarking the robustness of
visual representations in the context of object assembly tasks. Specifically,
it investigates the alignment and insertion of objects with geometrical
extrusions and intrusions, commonly referred to as a peg-in-hole task. The
accuracy required to detect and orient the peg and the hole geometry in SE(3)
space for successful assembly poses significant challenges. Addressing this, we
employ a general framework in visuomotor policy learning that utilizes visual
pretraining models as vision encoders. Our study investigates the robustness of
this framework when applied to a dual-arm manipulation setup, specifically to
the grasp variations. Our quantitative analysis shows that existing pretrained
models fail to capture the essential visual features necessary for this task.
However, a visual encoder trained from scratch consistently outperforms the
frozen pretrained models. Moreover, we discuss rotation representations and
associated loss functions that substantially improve policy learning. We
present a novel task scenario designed to evaluate the progress in visuomotor
policy learning, with a specific focus on improving the robustness of intricate
assembly tasks that require both geometrical and spatial reasoning. Videos,
additional experiments, dataset, and code are available at
https://bit.ly/geometric-peg-in-hole .
