  Pretrained representations from large-scale vision models have boosted the
performance of downstream embodied policy learning. We look to understand
whether additional self-supervised pretraining on exploration trajectories can
build on these general-purpose visual representations to better support
embodied planning in realistic environments. We evaluated four common auxiliary
losses in embodied AI, two hindsight-based losses, and a standard imitation
learning loss, by pretraining the agent's visual compression module and state
belief representations with each objective and using CLIP as a representative
visual backbone. The learned representations are then frozen for downstream
multi-step evaluation on two goal-directed tasks. Surprisingly, we find that
imitation learning on these exploration trajectories out-performs all other
auxiliary losses even despite the exploration trajectories being dissimilar
from the downstream tasks. This suggests that imitation of exploration may be
''all you need'' for building powerful planning representations. Additionally,
we find that popular auxiliary losses can benefit from simple modifications to
improve their support for downstream planning ability.
