  Current state-of-the-art detectors typically exploit feature pyramid to
detect objects at different scales. Among them, FPN is one of the
representative works that build a feature pyramid by multi-scale features
summation. However, the design defects behind prevent the multi-scale features
from being fully exploited. In this paper, we begin by first analyzing the
design defects of feature pyramid in FPN, and then introduce a new feature
pyramid architecture named AugFPN to address these problems. Specifically,
AugFPN consists of three components: Consistent Supervision, Residual Feature
Augmentation, and Soft RoI Selection. AugFPN narrows the semantic gaps between
features of different scales before feature fusion through Consistent
Supervision. In feature fusion, ratio-invariant context information is
extracted by Residual Feature Augmentation to reduce the information loss of
feature map at the highest pyramid level. Finally, Soft RoI Selection is
employed to learn a better RoI feature adaptively after feature fusion. By
replacing FPN with AugFPN in Faster R-CNN, our models achieve 2.3 and 1.6
points higher Average Precision (AP) when using ResNet50 and MobileNet-v2 as
backbone respectively. Furthermore, AugFPN improves RetinaNet by 1.6 points AP
and FCOS by 0.9 points AP when using ResNet50 as backbone. Codes will be made
available.
