  Multispectral pedestrian detection has attracted increasing attention from
the research community due to its crucial competence for many around-the-clock
applications (e.g., video surveillance and autonomous driving), especially
under insufficient illumination conditions. We create a human baseline over the
KAIST dataset and reveal that there is still a large gap between current top
detectors and human performance. To narrow this gap, we propose a network
fusion architecture, which consists of a multispectral proposal network to
generate pedestrian proposals, and a subsequent multispectral classification
network to distinguish pedestrian instances from hard negatives. The unified
network is learned by jointly optimizing pedestrian detection and semantic
segmentation tasks. The final detections are obtained by integrating the
outputs from different modalities as well as the two stages. The approach
significantly outperforms state-of-the-art methods on the KAIST dataset while
remain fast. Additionally, we contribute a sanitized version of training
annotations for the KAIST dataset, and examine the effects caused by different
kinds of annotation errors. Future research of this problem will benefit from
the sanitized version which eliminates the interference of annotation errors.
