  To achieve accurate and robust object detection in the real-world scenario,
various forms of images are incorporated, such as color, thermal, and depth.
However, multimodal data often suffer from the position shift problem, i.e.,
the image pair is not strictly aligned, making one object has different
positions in different modalities. For the deep learning method, this problem
makes it difficult to fuse multimodal features and puzzles the convolutional
neural network (CNN) training. In this article, we propose a general multimodal
detector named aligned region CNN (AR-CNN) to tackle the position shift
problem. First, a region feature (RF) alignment module with adjacent similarity
constraint is designed to consistently predict the position shift between two
modalities and adaptively align the cross-modal RFs. Second, we propose a novel
region of interest (RoI) jitter strategy to improve the robustness to
unexpected shift patterns. Third, we present a new multimodal feature fusion
method that selects the more reliable feature and suppresses the less useful
one via feature reweighting. In addition, by locating bounding boxes in both
modalities and building their relationships, we provide novel multimodal
labeling named KAIST-Paired. Extensive experiments on 2-D and 3-D object
detection, RGB-T, and RGB-D datasets demonstrate the effectiveness and
robustness of our method.
