  Correspondence learning is a fundamental problem in robotics, which aims to
learn a mapping between state, action pairs of agents of different dynamics or
embodiments. However, current correspondence learning methods either leverage
strictly paired data -- which are often difficult to collect -- or learn in an
unsupervised fashion from unpaired data using regularization techniques such as
cycle-consistency -- which suffer from severe misalignment issues. We propose a
weakly supervised correspondence learning approach that trades off between
strong supervision over strictly paired data and unsupervised learning with a
regularizer over unpaired data. Our idea is to leverage two types of weak
supervision: i) temporal ordering of states and actions to reduce the
compounding error, and ii) paired abstractions, instead of paired data, to
alleviate the misalignment problem and learn a more accurate correspondence.
The two types of weak supervision are easy to access in real-world
applications, which simultaneously reduces the high cost of annotating strictly
paired data and improves the quality of the learned correspondence.
