  Carefully curated and annotated datasets are the foundation of machine
learning, with particularly data-hungry deep neural networks forming the core
of what is often called Artificial Intelligence (AI). Due to the massive
success of deep learning applied to Earth Observation (EO) problems, the focus
of the community has been largely on the development of ever-more sophisticated
deep neural network architectures and training strategies largely ignoring the
overall importance of datasets. For that purpose, numerous task-specific
datasets have been created that were largely ignored by previously published
review articles on AI for Earth observation. With this article, we want to
change the perspective and put machine learning datasets dedicated to Earth
observation data and applications into the spotlight. Based on a review of the
historical developments, currently available resources are described and a
perspective for future developments is formed. We hope to contribute to an
understanding that the nature of our data is what distinguishes the Earth
observation community from many other communities that apply deep learning
techniques to image data, and that a detailed understanding of EO data
peculiarities is among the core competencies of our discipline.
