HongYiLee Backpropagation Notes

HongYiLee Backpropagation Notes

标签： Notes DeepLearning Backpropagation

---

Introduction of Deep Learning

1. Given a network structure, mean we define the function set.

给定一个神经网络的结构，那么就是定义了一个函数集合。然后我们要做的就是去找到一个最好的best function。

What does a Nerual network do?

$$y = f(x)=\sigma(W^L...\sigma(W^2(\sigma(W^1+b^1)+b^2)+...+b^L)$$
So we can using parallel computing techniques to speed up matrix operation.
对于矩阵运算，我们可以使用 GPU的加速运算能力。

2. Define the goodness or badness of a function. We call it Loss Function.

3. Pick the function minimize the Loss Function.

Backpropagation反向传播

$$L(\theta)=\sum_{n=1}^{N}l^n(\theta)$$
$$\frac{\partial L(\theta)}{\partial w} = \sum_{n=1}^{N}\frac{\partial l(\theta)}{\partial w}$$
So we need just to compute $\frac{\partial l(\theta)}{\partial w}$.
$$\frac{\partial l(\theta)}{\partial w} = \frac{\partial l(\theta)}{\partial z}\frac{\partial z}{\partial w}$$
So we have Forward pass:compute $\frac{\partial z}{\partial w}$.
and Backward pass: compute $\frac{\partial l(\theta)}{\partial z}$.

• Forward pass:
  $$\frac{\partial z}{\partial w_1}=x_1$$
  $$\frac{\partial z}{\partial w_2}=x_2$$
  $$...$$
  $$\frac{\partial z}{\partial w_i}=x_i$$
• Backward pass:
  Let $a=\sigma(z)$, $$\frac{\partial l}{\partial z}=\frac{\partial a}{\partial z}\frac{\partial l}{\partial a}$$
  $$\frac{\partial a}{\partial z}=\sigma^{'}(z)$$
  $$\frac{\partial l}{\partial a} = \frac{\partial z^{'}}{\partial a}\frac{\partial l}{\partial z^{'}} +\frac{\partial z^{''}}{\partial a}\frac{\partial l}{\partial z^{''}}$$ So this is Chain Rule.