  Pedestrian detection is used in many vision based applications ranging from
video surveillance to autonomous driving. Despite achieving high performance,
it is still largely unknown how well existing detectors generalize to unseen
data. This is important because a practical detector should be ready to use in
various scenarios in applications. To this end, we conduct a comprehensive
study in this paper, using a general principle of direct cross-dataset
evaluation. Through this study, we find that existing state-of-the-art
pedestrian detectors, though perform quite well when trained and tested on the
same dataset, generalize poorly in cross dataset evaluation. We demonstrate
that there are two reasons for this trend. Firstly, their designs (e.g. anchor
settings) may be biased towards popular benchmarks in the traditional
single-dataset training and test pipeline, but as a result largely limit their
generalization capability. Secondly, the training source is generally not dense
in pedestrians and diverse in scenarios. Under direct cross-dataset evaluation,
surprisingly, we find that a general purpose object detector, without
pedestrian-tailored adaptation in design, generalizes much better compared to
existing state-of-the-art pedestrian detectors. Furthermore, we illustrate that
diverse and dense datasets, collected by crawling the web, serve to be an
efficient source of pre-training for pedestrian detection. Accordingly, we
propose a progressive training pipeline and find that it works well for
autonomous-driving oriented pedestrian detection. Consequently, the study
conducted in this paper suggests that more emphasis should be put on
cross-dataset evaluation for the future design of generalizable pedestrian
detectors. Code and models can be accessed at
https://github.com/hasanirtiza/Pedestron.
