  FPN is a common component used in object detectors, it supplements
multi-scale information by adjacent level features interpolation and summation.
However, due to the existence of nonlinear operations and the convolutional
layers with different output dimensions, the relationship between different
levels is much more complex, the pixel-wise summation is not an efficient
approach. In this paper, we first analyze the design defects from pixel level
and feature map level. Then, we design a novel parameter-free feature pyramid
networks named Dual Refinement Feature Pyramid Networks (DRFPN) for the
problems. Specifically, DRFPN consists of two modules: Spatial Refinement Block
(SRB) and Channel Refinement Block (CRB). SRB learns the location and content
of sampling points based on contextual information between adjacent levels. CRB
learns an adaptive channel merging method based on attention mechanism. Our
proposed DRFPN can be easily plugged into existing FPN-based models. Without
bells and whistles, for two-stage detectors, our model outperforms different
FPN-based counterparts by 1.6 to 2.2 AP on the COCO detection benchmark, and
1.5 to 1.9 AP on the COCO segmentation benchmark. For one-stage detectors,
DRFPN improves anchor-based RetinaNet by 1.9 AP and anchor-free FCOS by 1.3 AP
when using ResNet50 as backbone. Extensive experiments verifies the robustness
and generalization ability of DRFPN. The code will be made publicly available.
