深度学习Deep learning小白入门笔记——学习李宏毅大佬视频笔记 20230708~20230710

Deep learning 2023/07/08~2023/07/10

Backpropagation

• $$\frac{\partial C}{\partial w}\to \frac{\partial z}{\partial w}\frac{\partial C}{\partial z}$$

  • Forward pass :

  $$\frac{\partial z}{\partial w}\text{\hspace{1em}(1)}$$

  Compute (1) for all parameters
  $$\frac{\partial z}{\partial w1}=?\text{\hspace{1em}(1a)}$$

  $$\frac{\partial z}{\partial w2}=?\text{\hspace{1em}(1b)}$$

  The value of the input connected by the weight.

  • Backward pass :

$$\frac{\partial C}{\partial z}\text{\hspace{1em}(2)}$$

​ Compute (2) for all activation function inputs z.

​ Compute (2) from the output layer.
$$\frac{\partial C}{\partial z}=\frac{\partial a}{\partial z}\frac{\partial C}{\partial a}\text{\hspace{1em}(2a)}$$

$$\frac{\partial C}{\partial z}=\frac{\partial z^{\prime }}{\partial a}\frac{\partial C}{\partial z^{\prime }}+\frac{\partial z^{\prime \prime }}{\partial a}\frac{\partial C}{\partial z^{\prime \prime }}\text{\hspace{1em}(2b)}$$

• Backpropagation Summary

$$ForwardPass\Rightarrow \frac{\partial z}{\partial w}=a$$

$$BackwardPass\Rightarrow \frac{\partial C}{\partial z}$$

$$ForwardPass\ast BackwardPass\Rightarrow \frac{\partial z}{\partial w}\frac{\partial C}{\partial z}=\frac{\partial C}{\partial w}$$

Regression

• Stock Market Forecast.
• Self-driving Car
• Recommendation
• Estimating the Combat Power(CP) of a pokemon after evolution

Estimating the Combat Power(CP) of a pokemon after evolution

• Step 1: Model

• Step 2: Goodness of Function

  • Loss function L:

    • Input: a function, output: how bad it is
      $$L(f)=L(w,b)$$

  • $$y=b+\sum w_i{x}_i$$

    $$L=\sum _n$$

• Step 3: Best Function & Gradient Descent

  • Best Function
    $$f^{\ast }=arg\underset{f}{\min }L(f)$$

    $$w^{\ast },b^{\ast }=argmi{n}_{w,b}L(w,b)$$

  • Gradient Descent

    • Consider loss function L(w) with one parameter w:
      $$w^{\ast }=arg\underset{w}{\min }L(w)$$

      • Pick an initial value w^0

      • Compute (1)
        $$\frac{dL}{dw}{|}_{w=w^0}\text{\hspace{1em}(1)}$$

        $$w^1\leftarrow w^0-\eta \frac{dL}{dw}{|}_{w=w^0}$$

        • η is called “learning rate”

      • Compute (2)
        $$\frac{dL}{dw}{|}_{w=w^1}\text{\hspace{1em}(2)}$$

        • next

        $$w^2\leftarrow w^1-\eta \frac{dL}{dw}{|}_{w=w^1}$$

    • How about two parameters ?
      $$w^{\ast },b^{\ast }=arg\underset{w,b}{\min }L(w,b)$$

  • Worry

    • In liner regression the loss function L is convex. (No local optimal)

  • A more complex model dose not always lead to better performance on testing data

    • This is Overfitting

• How to do Classification

  • Training data for Classification

  • Classification as Regression ?

    • Binary classification as example ?
    • Training: Class 1 means the target is 1; Class 2 means the target is -1.
    • Testing: closer to 1 → class 1; closer to -1 → class 2

  • Ideal Alternatives

    • Function (Model):
      $$x\Rightarrow f(x)=\{\begin{array}{l}g(x)>0Output=class1\\ elseOutput=class2\end{array}$$

• Loss function:
  $$L(f)=\sum _n\delta (f(x^n))\ne {\hat{y}}^n$$
  The number of times f get incorrect results on training data.

• Find the best function:

  • Example: Perceptron, SVM

• Gaussian Distribution
  f u , Σ ( x ) = 1 ( 2 π ) D / 2 1 ∣ Σ ∣ 1 / 2 e x p { − 1 2 ( x − μ ) T Σ − 1 ( x − μ ) } f_{u,\Sigma}(x)=\frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma|^{1/2}}exp\{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)\} fu,Σ​(x)=(2π)D/21​∣Σ∣1/21​exp{−21​(x−μ)TΣ−1(x−μ)}

  • Input: vector x, output: probability of sampling x.
    • The shape of the function determines by mean μ and covariance matrix ∑

• Maximun Likelihood
  f u , Σ ( x ) = 1 ( 2 π ) D / 2 1 ∣ Σ ∣ 1 / 2 e x p { − 1 2 ( x − μ ) T Σ − 1 ( x − μ ) } f_{u,\Sigma}(x)=\frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma|^{1/2}}exp\{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)\} fu,Σ​(x)=(2π)D/21​∣Σ∣1/21​exp{−21​(x−μ)TΣ−1(x−μ)}
  The Gaussian with any mean μ and covariance matrix ∑ can generate these points.
  $$x\Rightarrow P(C_1|x)=\frac{P(x|C_1)P(C_1)}{P(x|C_1)P(C_1)+P(x|C_2)P(C_2)}$$

• Three Steps

  • Function Set(Model) :

  $$x\Rightarrow \{\begin{array}{l}ifP(C_1|x)>0.5,output:class1\\ Otherwise,output:class2\end{array}$$

  • Goodness of a function
    • The mean μ and covariance ∑ that maximzing the likelihood(the probability of generating data)
    • Find the best function: easy

• Posterior Probability

$$P(C_1|x)=\sigma (z)sigmoidz=ln\frac{P(x|C_1)P(C_1)}{P(x|C_2)P(C_2)}$$

$$z=ln\frac{P(x|C_1)}{P(x|C_2)}+ln\frac{P(C_1)}{P(C_2)}\Rightarrow \frac{\frac{N_1}{N_1+N_2}}{\frac{N_2}{N_1+N_2}}=\frac{N_1}{N_2}$$

P ( x ∣ C 1 ) = 1 ( 2 π ) D / 2 1 ∣ Σ 1 ∣ 1 / 2 e x p { − 1 2 ( x − μ 1 ) T ( Σ 1 ) − 1 ( x − μ 1 ) } (3a) P(x|C_1)=\frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma^1|^{1/2}}exp\{-\frac{1}{2}(x-\mu^1)^T (\Sigma^{1})^{-1}(x-\mu^1)\} \tag{3a} P(x∣C1​)=(2π)D/21​∣Σ1∣1/21​exp{−21​(x−μ1)T(Σ1)−1(x−μ1)}(3a)

P ( x ∣ C 2 ) = 1 ( 2 π ) D / 2 1 ∣ Σ 2 ∣ 1 / 2 e x p { − 1 2 ( x − μ 2 ) T ( Σ 2 ) − 1 ( x − μ 2 ) } (3b) P(x|C_2)=\frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma^2|^{1/2}}exp\{-\frac{1}{2}(x-\mu^2)^T (\Sigma^{2})^{-1}(x-\mu^2)\} \tag{3b} P(x∣C2​)=(2π)D/21​∣Σ2∣1/21​exp{−21​(x−μ2)T(Σ2)−1(x−μ2)}(3b)

• According to (3a) and (3b), it is derived as follows

l n 1 ( 2 π ) D / 2 1 ∣ Σ 1 ∣ 1 / 2 e x p { − 1 2 ( x − μ 1 ) T ( Σ 1 ) − 1 ( x − μ 1 ) } 1 ( 2 π ) D / 2 1 ∣ Σ 2 ∣ 1 / 2 e x p { − 1 2 ( x − μ 2 ) T ( Σ 2 ) − 1 ( x − μ 2 ) } ln\frac{\frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma^1|^{1/2}}exp\{-\frac{1}{2}(x-\mu^1)^T (\Sigma^{1})^{-1}(x-\mu^1)\}}{\frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma^2|^{1/2}}exp\{-\frac{1}{2}(x-\mu^2)^T (\Sigma^{2})^{-1}(x-\mu^2)\}} ln(2π)D/21​∣Σ2∣1/21​exp{−21​(x−μ2)T(Σ2)−1(x−μ2)}(2π)D/21​∣Σ1∣1/21​exp{−21​(x−μ1)T(Σ1)−1(x−μ1)}​

---