为什么神经网络不能全零初始化

为什么神经网络不能全零初始化

相信所有学习过神经网络的人都知道神经网络的权重和偏置不能0初始化，但并不是所有人都知道为什么，在这里我们通过举例子+数学推导的方式解释原因。

假设要学习的神经网络结构如下所示：

初始化隐层参数为
$$\begin{gathered} {\mathbf{W}}^{(1)}=\left[\begin{array}{ccc}{w}_{11}^{(1)}& w_{12}^{(1)}& w_{13}^{(1)}\\ w_{21}^{(1)}& w_{22}^{(1)}& w_{23}^{(1)}\end{array}\right]=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\end{array}\right] \\ {\mathbf{b}}^{(1)}={\left[\begin{array}{cc}{b}_1^{(1)}& b_2^{(1)}\end{array}\right]}^T={\left[\begin{array}{cc}0& 0\end{array}\right]}^T \end{gathered}$$
同理，输出层参数为
$$\begin{gathered} {\mathbf{W}}^{(2)}=\left[\begin{array}{cc}{w}_{11}^{(2)}& w_{12}^{(2)}\end{array}\right]=\left[\begin{array}{cc}0& 0\end{array}\right] \\ {\mathbf{b}}^{(2)}={\left[\begin{array}{c}{b}_1^{(2)}\end{array}\right]}^T={\left[\begin{array}{c}0\end{array}\right]}^T \end{gathered}$$
隐层得到的输入和隐层的输出为
$$\begin{gathered} {\mathbf{z}}^{(1)}={\mathbf{W}}^{(1)}\mathbf{x}+{\mathbf{b}}^{(1)} \\ {\mathbf{a}}^{(1)}=f({\mathbf{z}}^{(1)}) \end{gathered}$$
其中$f$为激活函数，为方便之后的计算，写成标量形式
$$\begin{gathered} z_1^{(1)}=w_{11}^{(1)}{x}_1+w_{12}^{(1)}{x}_2+w_{13}^{(1)}{x}_3+b_1^{(1)} \\ {z}_1^{(2)}=w_{21}^{(1)}{x}_1+w_{22}^{(1)}{x}_2+w_{23}^{(1)}{x}_3+b_2^{(1)} \\ {a}_1^{(1)}=f(z_1^{(1)}) \\ {a}_2^{(1)}=f(z_2^{(1)}) \end{gathered}$$
输出层的输出为
$$\hat{y}=w_{11}^{(2)}{a}_1^{(1)}+w_{12}^{(2)}{a}_2^{(1)}+b_1^{(2)}$$
损失函数记为$L(y,\mathbf{W},\mathbf{b})$

由于神经网络初始化为0，所以
$$\begin{gathered} z_1^{(1)}=z_2^{(1)}=0 \\ {a}_1^{(1)}=a_2^{(1)} \\ \hat{y}=0 \end{gathered}$$
令
$$\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial \hat{y}}=\sigma$$
第一次反向传播
$$\begin{gathered} \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{11}^{(2)}}=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w_{11}^{(2)}}=\sigma a_1^{(1)} \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{12}^{(2)}}=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial w_{12}^{(2)}}=\sigma a_2^{(1)} \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial b_1^{(2)}}=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial b_1^{(2)}}=\sigma \end{gathered}$$
显然损失函数对$w_{11}^{(2)}$和$w_{12}^{(2)}$的偏导数相同，因此一次更新后两者仍然相同。这里不妨假设更新后不为0。

对隐层求偏导
$$\begin{gathered} \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{11}^{(1)}}=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial a_1^{(1)}}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})x_1=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{12}^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})x_2=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{13}^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})x_3=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial b_1^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{21}^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})x_1=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{22}^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})x_2=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{23}^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})x_3=0 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial b_2^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})=0 \end{gathered}$$
因为偏导都为0，所以第一次反向传播隐层参数不更新，仍然为0。

第二次正向传播时，由于隐层参数为0，所以
$$\begin{gathered} z_1^{(1)}=z_2^{(1)}=0 \\ {a}_1^{(1)}=a_2^{(1)} \end{gathered}$$
第二次反向传播
$$\begin{gathered} \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{11}^{(2)}}=\sigma a_1^{(1)} \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{12}^{(2)}}=\sigma a_2^{(1)} \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial b_1^{(2)}}=\sigma \end{gathered}$$
损失函数对$w_{11}^{(2)}$和$w_{12}^{(2)}$的偏导数相同，更新后两者仍然相同。

对隐层
$$\begin{gathered} \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{11}^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})x_1=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{21}^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})x_1 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{12}^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})x_2=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{22}^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})x_2 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{13}^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})x_3=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial w_{23}^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)})x_3 \\ \frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial b_1^{(1)}}=\sigma w_{11}^{(2)}{f}^{\prime }(z_1^{(1)})=\frac{\partial L(y,\mathbf{W},\mathbf{b})}{\partial b_2^{(1)}}=\sigma w_{12}^{(2)}{f}^{\prime }(z_2^{(1)}) \end{gathered}$$
那么更新后
$$\begin{gathered} {\mathbf{w}}_{1.}^{(1)}={\mathbf{w}}_{2.}^{(1)} \\ {b}_1^{(1)}=b_2^{(1)} \end{gathered}$$
其实就是隐层所有神经元的参数都相同了。

可以预见的是，由于隐层神经元参数都相同，那么在之后
$$\begin{gathered} z_1^{(1)}=z_2^{(1)} \\ {a}_1^{(1)}=a_2^{(1)} \end{gathered}$$
反向传播时，仍会出现输出层两个权值相同，隐层神经元参数相同。

换句话说，隐层神经元仅相当于一个节点，这可能直接导致模型失效！