深度学习（Deep Learning） 2.BP算法

深度学习（Deep Learning） 2.BP算法

---

1 Gradient Descent

• 初始化参数$\theta _0$
• 计算$\nabla L(\theta_0)$，$\theta_1 = \theta_0 - \eta\nabla L(\theta_0)$……
• 神经网络中参数巨多！

---

2 Chain Rule 链式法则

• $y=g(x),z=h(y)$
  $\frac{dz}{dx} = \frac{dz}{dy} +\frac{dy}{dx}$
• $x=g(s),y=h(s),z=k(x,y)$
  $\frac{dz}{ds} = \frac{\partial z}{\partial x}\frac{dx}{ds}+ \frac{\partial z}{\partial y}\frac{dy}{ds}$

---

3 BackPropagation

• $L(\theta) = \sum \limits_{n}C^n(\theta)$
• $\frac{\partial L(\theta)}{\partial w} = \sum \limits_{n} \frac{\partial C^n(\theta)}{\partial w}$
• $\frac{\partial C}{\partial z} = \frac{\partial C}{\partial a}\frac{\partial a}{\partial z}$，$\frac{\partial C}{\partial a} = \frac{\partial z'}{\partial a}\frac{\partial C}{\partial z'} + \frac{\partial C}{\partial z''}\frac{\partial z''}{\partial a}$，$a=\sigma(z)$
• $\frac{\partial C}{\partial z} = \sigma'(z)(w_3\frac{\partial C}{\partial z'}+w_4\frac{\partial a}{\partial z})$

---

4 BP算法推导

  给定训练集$D = \{(x_1,y_1),(x_2,y_2),…,(x_m,y_m)\},x_i \in R^d,y_i \in R^l$，输入层d个神经元，输出层l个，隐藏层q个。输出层第j个神经元的阈值为$\theta_j$，隐藏层第h个神经元的阈值为$\gamma_h$，

  对于训练实例$(x_k,y_k)$，假设其输出为$\hat y_k = (\hat y_1^k,\hat y_2^k,…,\hat y_l^k)$，即$\hat y_j^k = f(\beta_j-\theta_j)$，误差为$E_k = \frac{1}{2}\sum \limits_{j=1}^l(\hat y_j^k - y_j^k)^2$

网络中总共有$(d+l+1)q + l$个参数需要确定：输入层到隐藏层的$d*q$个，隐藏层到输出层的$q*l$个，$q$个隐藏层神经元阈值，$l$个输出层神经元阈值

• 隐藏层到输出层的权值$w_{hj}$的推导：$\nabla w_{hj}= -\eta \frac{\partial E_k}{\partial w_{hj}}$
  $\frac{\partial E_k}{\partial w_{hj}} = \frac{\partial E_k}{\partial \hat y_j^k} \frac{\partial \hat y_j^k}{\partial \beta_j} \frac{\partial \beta_j }{\partial w_{hj}}$，且$\frac{\beta_j }{\partial w_{hj}}= b_h$
  sigmoid函数的性质有：$f'(x) = f(x)(1-f(x))$
  $g_j = - \frac{\partial E_k}{\partial \hat y_j^k} \frac{\partial \hat y_j^k}{\partial \beta_j} = -(\hat y_j^k - y_j^k)f'(\beta_j - \theta_j) = \hat y_j^k(1-\hat y_j^k)(y_j^k -\hat y_j^k)$
  因此$\nabla w_{hj} = \eta g_j b_h$
• 类似可以得到：
  $\nabla \theta_j = -\eta g_j$
  $\nabla v_{ih} = \eta e_h x_i$
  $\nabla \gamma_h = -\eta e_h$
  其中$e_h = - \frac{\partial E_k}{\partial b_h} \frac{\partial b_h}{\partial \alpha_h} = -\sum\limits_{j=1}^l \frac{\partial E_k}{\partial \beta_j} \frac{\partial \beta_j}{\partial b_h}f'(\alpha_h- \gamma_h) = \sum\limits_{j=1}^l w_{hj} g_j f'(\alpha_h- \gamma_h) = b_h(1-b_h)\sum\limits_{j=1}^l w_{hj} g_j$