depth-wise convolution and depth-wise separable convolution

Introduction

The standard convolution layer involve $(channe{l}_{input},channe{l}_{output},filter\_size)$,for instance,
$$Input-channel:10$$
$$Output-channel:20$$
$$Filter-size:7$$
可以得到$Parameters=(7\times 7\times 10+1)\times 20=9820$
the amount of parameters of the standard convolution is so much that the model is more probably over-fitting.So the depth-wise convolution and the depth-wise separable convolution is proposed to avoid this scenarios.

depth-wise convolution

We use each filter channels only at one input channels.

To produce the same effect with normal convolution, what we need to do is select a channel ,make all the elements zeros in the filter except that channel and then convolve.
Although the parameters remain same , depth-wise convolution can produce 3 output channels wth only one 3-channel filter, but the standard convolution produce only 1 output channel with the same filter.

depth-wise separable convolution

We perform depth-wise convolution at horizontal dimension( height and width ) and after that we use $1\times 1$convolution to cover the depth dimension so that we can produce any channels we want.

Parameter:
Requirements:
$$Input-channel:3$$
$$Output-channel:3$$
$$filter-size:3$$
the standard convolution: $Parameters=(3\times 3\times 3+1)\times 3=84$
depth-wise separable convolution : $Parameters=(3\times 3+1)\times 3+3\times 3=39$

Having too many parameters forces function to memorize lather than learn and thus over-fitting.Depth-wise separable convolution save us from that.