  Visual representation learning hold great promise for robotics, but is
severely hampered by the scarcity and homogeneity of robotics datasets. Recent
works address this problem by pre-training visual representations on
large-scale but out-of-domain data (e.g., videos of egocentric interactions)
and then transferring them to target robotics tasks. While the field is heavily
focused on developing better pre-training algorithms, we find that dataset
choice is just as important to this paradigm's success. After all, the
representation can only learn the structures or priors present in the
pre-training dataset. To this end, we flip the focus on algorithms, and instead
conduct a dataset centric analysis of robotic pre-training. Our findings call
into question some common wisdom in the field. We observe that traditional
vision datasets (like ImageNet, Kinetics and 100 Days of Hands) are
surprisingly competitive options for visuo-motor representation learning, and
that the pre-training dataset's image distribution matters more than its size.
Finally, we show that common simulation benchmarks are not a reliable proxy for
real world performance and that simple regularization strategies can
dramatically improve real world policy learning.
https://data4robotics.github.io
