  Existing RGB-D salient object detection (SOD) approaches concentrate on the
cross-modal fusion between the RGB stream and the depth stream. They do not
deeply explore the effect of the depth map itself. In this work, we design a
single stream network to directly use the depth map to guide early fusion and
middle fusion between RGB and depth, which saves the feature encoder of the
depth stream and achieves a lightweight and real-time model. We tactfully
utilize depth information from two perspectives: (1) Overcoming the
incompatibility problem caused by the great difference between modalities, we
build a single stream encoder to achieve the early fusion, which can take full
advantage of ImageNet pre-trained backbone model to extract rich and
discriminative features. (2) We design a novel depth-enhanced dual attention
module (DEDA) to efficiently provide the fore-/back-ground branches with the
spatially filtered features, which enables the decoder to optimally perform the
middle fusion. Besides, we put forward a pyramidally attended feature
extraction module (PAFE) to accurately localize the objects of different
scales. Extensive experiments demonstrate that the proposed model performs
favorably against most state-of-the-art methods under different evaluation
metrics. Furthermore, this model is 55.5\% lighter than the current lightest
model and runs at a real-time speed of 32 FPS when processing a $384 \times
384$ image.
