李宏毅机器学习（8）

Backpropagation

Gradient Descent

Network parameters θ = { w 1 , w 2 , . . . , b 1 , b 2 , . . . } θ=\{w_1,w_2,...,b_1,b_2,...\} θ={w1​,w2​,...,b1​,b2​,...}

尽管理论上可以直接计算，但Network里参数太多，实际运用中难以计算。
解决方案：Backpropagation算法。
前置技能：Chain Rule 链式法则


$L(\theta )={\sum }_{n=1}^N{C}_n(\theta )\to \frac{\alpha L(\theta )}{\alpha w}={\sum }_{n=1}^N\frac{\alpha C_n(\theta )}{\alpha w}$
例：

$\frac{\alpha C}{\alpha w}=\frac{\alpha z}{\alpha w}\frac{\alpha c}{\alpha z}$
Forward pass:Compute $\frac{\alpha z}{\alpha w}$ for all parameters
Backward pass:Compute $\frac{\alpha c}{\alpha z}$ for all $z$

1. Forward pass:$\frac{\alpha z}{\alpha w_1}=x_1,\frac{\alpha z}{\alpha w_2}=x_2$，容易计算
2. Backward pass:
   

Case 1:

$\frac{\alpha c}{\alpha z}=\frac{\alpha a}{\alpha z}\frac{\alpha c}{\alpha a}={\sigma }^{{}^{\prime }}(z)\frac{\alpha c}{\alpha a}$
$\frac{\alpha c}{\alpha a}=\frac{\alpha z^{\prime }}{\alpha a}\frac{\alpha c}{\alpha z^{\prime }}+\frac{\alpha z^{\prime \prime }}{\alpha a}\frac{\alpha c}{\alpha z^{\prime \prime }}=w_3\frac{\alpha c}{\alpha z^{\prime }}+w_4\frac{\alpha c}{\alpha z^{\prime \prime }}$，假设已经知道了$w_3$和$w_4$
$\therefore \frac{\alpha c}{\alpha z}={\sigma }^{\prime }(z)[w_3\frac{\alpha c}{\alpha z^{\prime }}+w_4\frac{\alpha c}{\alpha z^{\prime \prime }}]$
再次展开：$\frac{\alpha c}{\alpha z^{\prime }}=\frac{\alpha y_1}{\alpha z^{\prime }}\frac{\alpha c}{\alpha y_1}$,$\frac{\alpha c}{\alpha z^{\prime \prime }}=\frac{\alpha y_2}{\alpha z^{\prime \prime }}\frac{\alpha c}{\alpha y_2}$

Case 2:

继续用Case 1的方法递归计算即可，知道结尾。

总结

就是用连式法则不断递归运算。