  Multi-scale features are essential for dense prediction tasks, such as object
detection, instance segmentation, and semantic segmentation. The prevailing
methods usually utilize a classification backbone to extract multi-scale
features and then fuse these features using a lightweight module (e.g., the
fusion module in FPN and BiFPN, two typical object detection methods). However,
as these methods allocate most computational resources to the classification
backbone, the multi-scale feature fusion in these methods is delayed, which may
lead to inadequate feature fusion. While some methods perform feature fusion
from early stages, they either fail to fully leverage high-level features to
guide low-level feature learning or have complex structures, resulting in
sub-optimal performance. We propose a streamlined cascade encoder-decoder
network, dubbed CEDNet, tailored for dense \mbox{prediction} tasks. All stages
in CEDNet share the same encoder-decoder structure and perform multi-scale
feature fusion within the decoder. A hallmark of CEDNet is its ability to
incorporate high-level features from early stages to guide low-level feature
learning in subsequent stages, thereby enhancing the effectiveness of
multi-scale feature fusion. We explored three well-known encoder-decoder
structures: Hourglass, UNet, and FPN. When integrated into CEDNet, they
performed much better than traditional methods that use a pre-designed
classification backbone combined with a lightweight fusion module. Extensive
experiments on object detection, instance segmentation, and semantic
segmentation demonstrated the effectiveness of our method. The code is
available at https://github.com/zhanggang001/CEDNet.
