  The field of visual representation learning has seen explosive growth in the
past years, but its benefits in robotics have been surprisingly limited so far.
Prior work uses generic visual representations as a basis to learn
(task-specific) robot action policies (e.g., via behavior cloning). While the
visual representations do accelerate learning, they are primarily used to
encode visual observations. Thus, action information has to be derived purely
from robot data, which is expensive to collect! In this work, we present a
scalable alternative where the visual representations can help directly infer
robot actions. We observe that vision encoders express relationships between
image observations as distances (e.g., via embedding dot product) that could be
used to efficiently plan robot behavior. We operationalize this insight and
develop a simple algorithm for acquiring a distance function and dynamics
predictor, by fine-tuning a pre-trained representation on human collected video
sequences. The final method is able to substantially outperform traditional
robot learning baselines (e.g., 70% success v.s. 50% for behavior cloning on
pick-place) on a suite of diverse real-world manipulation tasks. It can also
generalize to novel objects, without using any robot demonstrations during
train time. For visualizations of the learned policies please check:
https://agi-labs.github.io/manipulate-by-seeing/.
