  Earth observation (EO), aiming at monitoring the state of planet Earth using
remote sensing data, is critical for improving our daily lives and living
environment. With a growing number of satellites in orbit, an increasing number
of datasets with diverse sensors and research domains are being published to
facilitate the research of the remote sensing community. This paper presents a
comprehensive review of more than 500 publicly published datasets, including
research domains like agriculture, land use and land cover, disaster
monitoring, scene understanding, vision-language models, foundation models,
climate change, and weather forecasting. We systematically analyze these EO
datasets from four aspects: volume, resolution distributions, research domains,
and the correlation between datasets. Based on the dataset attributes, we
propose to measure, rank, and select datasets to build a new benchmark for
model evaluation. Furthermore, a new platform for EO, termed EarthNets, is
released to achieve a fair and consistent evaluation of deep learning methods
on remote sensing data. EarthNets supports standard dataset libraries and
cutting-edge deep learning models to bridge the gap between the remote sensing
and machine learning communities. Based on this platform, extensive
deep-learning methods are evaluated on the new benchmark. The insightful
results are beneficial to future research. The platform and dataset collections
are publicly available at https://earthnets.github.io.
