  Autonomous driving has rapidly developed and shown promising performance due
to recent advances in hardware and deep learning techniques. High-quality
datasets are fundamental for developing reliable autonomous driving algorithms.
Previous dataset surveys either focused on a limited number or lacked detailed
investigation of dataset characteristics. To this end, we present an exhaustive
study of 265 autonomous driving datasets from multiple perspectives, including
sensor modalities, data size, tasks, and contextual conditions. We introduce a
novel metric to evaluate the impact of datasets, which can also be a guide for
creating new datasets. Besides, we analyze the annotation processes, existing
labeling tools, and the annotation quality of datasets, showing the importance
of establishing a standard annotation pipeline. On the other hand, we
thoroughly analyze the impact of geographical and adversarial environmental
conditions on the performance of autonomous driving systems. Moreover, we
exhibit the data distribution of several vital datasets and discuss their pros
and cons accordingly. Finally, we discuss the current challenges and the
development trend of the future autonomous driving datasets.
