  The past years have witnessed great progress on remote sensing (RS) image
interpretation and its wide applications. With RS images becoming more
accessible than ever before, there is an increasing demand for the automatic
interpretation of these images. In this context, the benchmark datasets serve
as essential prerequisites for developing and testing intelligent
interpretation algorithms. After reviewing existing benchmark datasets in the
research community of RS image interpretation, this article discusses the
problem of how to efficiently prepare a suitable benchmark dataset for RS image
interpretation. Specifically, we first analyze the current challenges of
developing intelligent algorithms for RS image interpretation with bibliometric
investigations. We then present the general guidances on creating benchmark
datasets in efficient manners. Following the presented guidances, we also
provide an example on building RS image dataset, i.e., Million-AID, a new
large-scale benchmark dataset containing a million instances for RS image scene
classification. Several challenges and perspectives in RS image annotation are
finally discussed to facilitate the research in benchmark dataset construction.
We do hope this paper will provide the RS community an overall perspective on
constructing large-scale and practical image datasets for further research,
especially data-driven ones.
