单目图像深度估计算法-FastDepth

基于深度学习的单目深度估计在近几年是比较热门的研究方向之一，MIT的Diana Wofk等人在ICRA 2019上提出了一种用于嵌入式系统的深度估计算法FastDepth，在保证准确率的情况下，大大提高了模型的计算效率。

论文：FastDepth: Fast Monocular Depth Estimation on Embedded Systems
Offical Pytorch：https://github.com/dwofk/fast-depth
方法
模型

模型的整体结构比较简单，采用了Encoder-Decoder的架构。Encoder部分采用了MobileNet模型提取到7x7x1024的特征；Decoder部分采用了5次上采样，中间三次上采样结果通过Skip Connections的方法分别与Encoder部分的特征进行了特征融合，为了减小上采样部分的通道特征，还使用了5x5的卷积来降维；最后使用1*1的卷积得到深度图。
5164048-23eae66f8d5a002e.png
Model

使用Keras实现基本的FastDepth模型：

from keras.layers import Conv2D, UpSampling2D, SeparableConv2D, BatchNormalization, Activation, add
from keras.models import Model
from keras.applications.mobilenet import MobileNet
 
     
class FastDepth:
    def __init__(self):
        self.build_net()
 
    def _SDWConv(self, filtres, kernel):
        def f(x):
            x = SeparableConv2D(filtres, kernel, padding='same')(x)
            x = BatchNormalization()(x)
            x = Activation('relu')(x)
 
            return x
        return f
 
    def _encoder(self):
        self.MN = MobileNet(input_shape=(224, 224, 3),
                            weights=None,
                            include_top='False')
 
        # 7*7*1024
        latent = self.MN.get_layer('conv_pw_13_relu').output
 
        return latent
 
    def _decoder(self, x):
        # 14*14*512
        x1 = self._SDWConv(512, (5, 5))(x)
        x1 = UpSampling2D()(x1)
 
        # 28*28*256
        x2 = self._SDWConv(256, (5, 5))(x1)
        x2 = UpSampling2D()(x2)
        s2 = self.MN.get_layer('conv_pw_5_relu').output
        x2 = add([x2, s2])
 
        # 56*56*128
        x3 = self._SDWConv(128, (5, 5))(x2)
        x3 = UpSampling2D()(x3)
        s3 = self.MN.get_layer('conv_pw_3_relu').output
        x3 = add([x3, s3])
 
        # 112*112*64
        x4 = self._SDWConv(64, (5, 5))(x3)
        x4 = UpSampling2D()(x4)
        s4 = self.MN.get_layer('conv_pw_1_relu').output
        x4 = add([x4, s4])
 
        # 224*224*32
        x5 = self._SDWConv(32, (5, 5))(x4)
        x5 = UpSampling2D()(x5)
 
        return x5
 
    def build_net(self):
        latent = self._encoder()
        out = self._decoder(latent)
        out_dense = Conv2D(1, (1, 1))(out)
 
        self.model = Model(inputs=self.MN.input, outputs=out_dense)
 
 
if __name__ == '__main__':
    net = FastDepth()
    net.model.summary()

Decoder部分的结构如下所示：

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
 省略MobileNet...
__________________________________________________________________________________________________  
conv_pw_13_relu (ReLU)          (None, 7, 7, 1024)   0           conv_pw_13_bn[0][0]              
__________________________________________________________________________________________________
separable_conv2d_3 (SeparableCo (None, 7, 7, 512)    550400      conv_pw_13_relu[0][0]            
__________________________________________________________________________________________________
batch_normalization_1 (BatchNor (None, 7, 7, 512)    2048        separable_conv2d_3[0][0]         
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 7, 7, 512)    0           batch_normalization_1[0][0]      
__________________________________________________________________________________________________
up_sampling2d_3 (UpSampling2D)  (None, 14, 14, 512)  0           activation_1[0][0]               
__________________________________________________________________________________________________
separable_conv2d_4 (SeparableCo (None, 14, 14, 256)  144128      up_sampling2d_3[0][0]            
__________________________________________________________________________________________________
batch_normalization_2 (BatchNor (None, 14, 14, 256)  1024        separable_conv2d_4[0][0]         
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 14, 14, 256)  0           batch_normalization_2[0][0]      
__________________________________________________________________________________________________
up_sampling2d_4 (UpSampling2D)  (None, 28, 28, 256)  0           activation_2[0][0]               
__________________________________________________________________________________________________
add_2 (Add)                     (None, 28, 28, 256)  0           up_sampling2d_4[0][0]            
                                                                 conv_pw_5_relu[0][0]             
__________________________________________________________________________________________________
separable_conv2d_5 (SeparableCo (None, 28, 28, 128)  39296       add_2[0][0]                      
__________________________________________________________________________________________________
batch_normalization_3 (BatchNor (None, 28, 28, 128)  512         separable_conv2d_5[0][0]         
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 28, 28, 128)  0           batch_normalization_3[0][0]      
__________________________________________________________________________________________________
up_sampling2d_5 (UpSampling2D)  (None, 56, 56, 128)  0           activation_3[0][0]               
__________________________________________________________________________________________________
add_3 (Add)                     (None, 56, 56, 128)  0           up_sampling2d_5[0][0]            
                                                                 conv_pw_3_relu[0][0]             
__________________________________________________________________________________________________
separable_conv2d_6 (SeparableCo (None, 56, 56, 64)   11456       add_3[0][0]                      
__________________________________________________________________________________________________
batch_normalization_4 (BatchNor (None, 56, 56, 64)   256         separable_conv2d_6[0][0]         
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 56, 56, 64)   0           batch_normalization_4[0][0]      
__________________________________________________________________________________________________
up_sampling2d_6 (UpSampling2D)  (None, 112, 112, 64) 0           activation_4[0][0]               
__________________________________________________________________________________________________
add_4 (Add)                     (None, 112, 112, 64) 0           up_sampling2d_6[0][0]            
                                                                 conv_pw_1_relu[0][0]             
__________________________________________________________________________________________________
separable_conv2d_7 (SeparableCo (None, 112, 112, 32) 3680        add_4[0][0]                      
__________________________________________________________________________________________________
batch_normalization_5 (BatchNor (None, 112, 112, 32) 128         separable_conv2d_7[0][0]         
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 112, 112, 32) 0           batch_normalization_5[0][0]      
__________________________________________________________________________________________________
up_sampling2d_7 (UpSampling2D)  (None, 224, 224, 32) 0           activation_5[0][0]               
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 224, 224, 1)  33          up_sampling2d_7[0][0]            
==================================================================================================
Total params: 3,981,825
Trainable params: 3,957,953
Non-trainable params: 23,872
__________________________________________________________________________________________________

网络裁剪

为了减小模型体积，提高运算效率，使得模型更适用于嵌入式设备，使用NetAdapt算法对FastDepth进行了裁剪。