  Current methods in training and benchmarking vision models exhibit an
over-reliance on passive, curated datasets. Although models trained on these
datasets have shown strong performance in a wide variety of tasks such as
classification, detection, and segmentation, they fundamentally are unable to
generalize to an ever-evolving world due to constant out-of-distribution shifts
of input data. Therefore, instead of training on fixed datasets, can we
approach learning in a more human-centric and adaptive manner? In this paper,
we introduce Action-Aware Embodied Learning for Perception (ALP), an embodied
learning framework that incorporates action information into representation
learning through a combination of optimizing a reinforcement learning policy
and an inverse dynamics prediction objective. Our method actively explores in
complex 3D environments to both learn generalizable task-agnostic visual
representations as well as collect downstream training data. We show that ALP
outperforms existing baselines in several downstream perception tasks. In
addition, we show that by training on actively collected data more relevant to
the environment and task, our method generalizes more robustly to downstream
tasks compared to models pre-trained on fixed datasets such as ImageNet.
