(2021 ICCV)Specificity-preserving RGB-D Saliency Detection(A类)

# 一、作者

​ Tao Zhou ， Huazhu Fu ，Geng Chen ，Yi Zhou ，Deng-Ping Fan， Ling Shao

二、地址

2.1 原文地址

ICCV 地址

2.2 代码地址

代码

三、摘要

​ RGB-D saliency detection has attracted increasing attention, due to its effectiveness and the fact that depth cues can now be conveniently captured. Existing works often focus on learning a shared representation through various fusion strategies, with few methods explicitly considering how to preserve modality-specific characteristics. In this paper, taking a new perspective, we propose a specificity preserving network (SP-Net) for RGB-D saliency detection, which benefits saliency detection performance by exploring both the shared information and modality-specific properties (e.g., specificity). Specifically, two modality-specific networks and a shared learning network are adopted to generate individual and shared saliency maps. A cross enhanced integration module (CIM) is proposed to fuse cross-modal features in the shared learning network, which are then propagated to the next layer for integrating cross level information. Besides, we propose a multi-modal feature aggregation (MFA) module to integrate the modality specific features from each individual decoder into the shared decoder, which can provide rich complementary multi-modal information to boost the saliency detection performance. Further, a skip connection is used to combine hierarchical features between the encoder and decoder layers. Experiments on six benchmark datasets demonstrate that our SP-Net outperforms other state-of-the-art methods.

四、主要内容

4.1 主要工作

• We propose a novel specificity-preserving network for RGB-D saliency detection (SP-Net), which can explore the shared information as well as preserve modality-specific characteristics.

• We propose a cross-enhanced integration module (CIM) (互补互动模块) to fuse the cross-modal features and learn shared representations for the two modalities. The output of each CIM is then propagated to the next layer to capture cross-level information.

• We propose a simple but effective multi-modal feature aggregation (MFA) module to integrate these learned modality-specific features. It makes full use of the features learned in the modality-specific decoder to boost the saliency detection performance.

• Extensive experiments on six public datasets demonstrate the superiority of our model over thirty benchmarking methods. Moreover, we carry out an attributebased evaluation to study the performance of many state-of-the-art RGB-D saliency detection methods under different challenging factors (e.g., number of salient objects, indoor or outdoor environments, and light conditions), which has not been done previously by existing studies.

4.2 网路结构图

4.2.1 总体结构图 (Res2Net-50)

​ The overall architecture of the proposed SP-Net. Our model consists of two modality-specific learning networks and a shared learning network. The modality-specific learning networks are used to preserve the individual properties for each modality (i.e., RGB or depth), while the shared network is used to fuse cross-modal features and explore their complementary information. Skip connections are adopted to combine hierarchical features between the encoder and decoder layers. The learned features from the modality-specific decoder are integrated into the shared decoder to provide rich multi-modal complementary information for boosting saliency detection performance. Here, “C” denotes feature concatenation.

4.2.2 CIM (to learn shared feature representation)

4.2.3 MFA (to integrate features into the shared decoder)

4.3 LOSS

$$\begin{gathered} L_{total}=L_{sh}(S_{sh},G)+L_{sp}(S_R,G)+L_{sp}(S_D,G) \\ {L}_{sp}:modality-specific \\ {L}_{sh}:shareddecoders \\ {S}_R,S_D:predictionmapswhenusingRGBanddepthimages \\ {S}_{sh}:predectionmapusingtheirsharedrepresentation \\ G:groundtruthmap \end{gathered}$$

五、评估材料

​ S-measure、E_ϕ 、F-measure、MAE

$$\begin{gathered} S_{\alpha }=\alpha \ast S_o+(1-\alpha )\ast S_r \\ {S}_r:regionperception \\ {S}_o:objectperception \end{gathered}$$

$$E_{\varphi }=\frac{1}{W\ast H}\sum _{i=1}^W\sum _{j=1}^H\varphi FM(i,j)$$

$$F_{\beta }=(1+{\beta }^2)\frac{P\ast R}{{\beta }^{2}P+R}$$

六、结论

​ In this paper, we have proposed a novel SP-Net for RGB-D saliency detection. Different from most existing works, which mainly focus on learning shared representations, our model not only explores the shared cross-modal information but also compensates modality-specific characteristics to boost the saliency detection performance. Besides, the proposed CIM can propagate information across modalities and layers, while our MFA module can provide specific properties to the shared decoder to enhance the complementary multi-modal information. Quantitative and qualitative evaluations conducted on six challenging benchmark datasets demonstrate the superiority of our SP-Net over other existing RGB-D saliency detection approaches. In the future, we can apply our model to the light field saliency detection task .