  This paper presents the Rail-5k dataset for benchmarking the performance of
visual algorithms in a real-world application scenario, namely the rail surface
defects detection task. We collected over 5k high-quality images from railways
across China, and annotated 1100 images with the help from railway experts to
identify the most common 13 types of rail defects. The dataset can be used for
two settings both with unique challenges, the first is the fully-supervised
setting using the 1k+ labeled images for training, fine-grained nature and
long-tailed distribution of defect classes makes it hard for visual algorithms
to tackle. The second is the semi-supervised learning setting facilitated by
the 4k unlabeled images, these 4k images are uncurated containing possible
image corruptions and domain shift with the labeled images, which can not be
easily tackle by previous semi-supervised learning methods. We believe our
dataset could be a valuable benchmark for evaluating robustness and reliability
of visual algorithms.
