关于反向传播算法中几个公式的推导

** 参考自资料，[戳此处](http://deeplearning.stanford.edu/wiki/index.php/%E5%8F%8D%E5%90%91%E4%BC%A0%E5%AF%BC%E7%AE%97%E6%B3%95)**

$$J(W,b;x,y)=\frac{1}{2}||h_{W,b}(x)-y|{|}^2$$

符号说明：
$$\begin{gathered} \begin{array}{rlrl}l& =神经网络的某一层 \\ {S}_l& =第l层神经元的数目 \\ L& =神经网络的总层数\end{array} \end{gathered}$$
如下图中：
$$S_1=2;S_L=S_3=2$$

推导之前先列出几个用到的等式：

$$\begin{gathered} \begin{array}{rl}{z}_j^l& =\sum _{k=1}^{S_{l-1}}f(z_k^{l-1})\cdot W_{jk}^{l-1}\\ \\ & =f(z_1^{l-1})\cdot W_{j1}^{l-1}+f(z_2^{l-1})\cdot W_{j2}^{l-1}+\cdots +f(z_{S_{l-1}}^{l-1})\cdot W_{j{S}_{l-1}}^{l-1}\cdots \cdots \cdots (1)\\ \\ & =a_1^{l-1}\cdot W_{j1}^{l-1}+a_2^{l-1}\cdot W_{j2}^{l-1}+\cdots +a_{S_{l-1}}^{l-1}\cdot W_{j{S}_{l-1}}^{l-1}\cdots \cdots \cdots \cdots \cdots (2)\end{array} \end{gathered}$$

例如：

$$\begin{array}{r}{z}_1^2=\sum _{k=1}^{3}f(z_k^1)\cdot W_{1k}^1=a_1^1\cdot W_{11}^1+a_2^1\cdot W_{12}^1+a_3^1\cdot W_{13}^1\end{array}$$

$$\begin{array}{r}h(x_i)=a_i^L=f(z_i^L);z_i^l=\sum _{j=1}^{S_{(l-1)}}{W}_{ij}^{l-1}\cdot a_j^{l-1}\cdots \cdots \cdots \cdots (3)\end{array}$$

定义：${\delta }_i^l=\frac{\partial J}{\partial z_i^l}$；这只是定义 的一个称之为“残差”的中间变量

推导：

---

公式一：$\frac{\partial J}{\partial W_{ij}^l}={\delta }_i^{l+1}\cdot a_j^l$

$$\begin{gathered} \begin{array}{rl}\frac{\partial J}{\partial W_{ij}^l}& =\sum _{k=1}^{S_{l+1}}\frac{\partial J}{\partial z_k^{l+1}}\cdot \frac{\partial z_k^{l+1}}{\partial W_{ij}^l}\\ & \\ & =\frac{\partial J}{\partial z_i^{l+1}}\cdot \frac{\partial z_i^{l+1}}{\partial W_{ij}^l}\cdots \cdots 其中只有当k等于i时，\frac{\partial z_i^{l+1}}{\partial W_{ij}^l}才不为零\\ \\ & ={\delta }_i^{l+1}\cdot \frac{\partial z_i^{l+1}}{\partial W_{ij}^l}\\ \\ & ={\delta }_i^{l+1}\cdot \frac{\partial }{\partial W_{ij}^l}[a_1^l{W}_{i1}^l+a_2^l{W}_{i2}^l+\cdots +a_1^l{W}_{i{S}_l}^l]\cdots \cdots \\ \\ & \\ & \end{array} \end{gathered}$$