7月25日深度学习笔记——Classification

---

前言

本文为7月25日深度学习笔记，分为三个章节：

• How to do Classification；
• Logistic Regression；
• Intro to Deep Learning.

---

一、How to do Classification

1、Two Classes

蓝球来自于$B_1$的概率：
$$P(B_1|Blue)=\frac{P(Blue|B_1)P(B_1)}{P(Blue|B_1)P(B_1)+P(Blue|B_2)P(B_2)}$$

Given an $x$, which class does it belong to:
$$P(C_1|x)=\frac{P(x|C_1)P(C_1)}{P(x|C_1)P(C_1)+P(x|C_2)P(C_2)}$$

2、Gaussian Distribution

f μ , ∑ ( x ) = 1 ( 2 π D / 2 ) 1 ∣ ∑ ∣ 1 / 2 e x p { − 1 2 ( x − μ ) T ∑ − 1 ( x − μ ) } f_{\mu, \sum}(x) = \frac{1}{(2\pi ^{D/2})} \frac{1}{|\sum|^{1/2}} exp\{-\frac {1}{2}(x-\mu)^T \textstyle \sum^{-1}(x-\mu) \} fμ,∑​(x)=(2πD/2)1​∣∑∣1/21​exp{−21​(x−μ)T∑−1(x−μ)}

• input: vector x;

• output: probability of sampling x.

  Where $\mu$ is the mean, $\sum$ is the covariance matirx（协方差矩阵）.

3、Maximum Likelihood

$$\begin{gathered} L(\mu ,\sum )=f_{\mu ,\sum }(x^1)f_{\mu ,\sum }(x^2)\dots f_{\mu ,\sum }(x^n) \\ \mu \ast ,\sum \ast =argminL(\mu ,\sum ) \\ \mu \ast =\frac{1}{79}\sum _{n=1}^n{x}^n\sum \ast =\frac{1}{79}\sum _{n=1}^n(x^n-\mu \ast )(x^n-\mu \ast {)}^T \end{gathered}$$

P ( C 1 ∣ x ) = P ( x ∣ C 1 ) P ( C 1 ) P ( x ∣ C 1 ) P ( C 1 ) + P ( x ∣ C 2 ) P ( C 2 ) P ( x ∣ C 1 ) = f μ 1 , ∑ 1 ( x ) = 1 ( 2 π D / 2 ) 1 ∣ ∑ 1 ∣ 1 / 2 e x p { − 1 2 ( x − μ 1 ) T ( ∑ 1 ) − 1 ( x − μ 1 ) } P ( C 1 ) = 0.56 P ( x ∣ C 2 ) = f μ 2 , ∑ 2 ( x ) P ( C 2 ) = 0.44 P(C_1|x) = \frac{P(x|C_1) P(C_1)}{P(x|C_1) P(C_1) + P(x|C_2) P(C_2)}\\ P(x|C_1) = f_{\mu^1, \sum^1}(x) = \frac{1}{(2\pi ^{D/2})} \frac{1}{|\sum^1|^{1/2}} exp\{-\frac {1}{2}(x-\mu^1)^T \textstyle (\sum^1)^{-1}(x-\mu^1) \}\\ P(C_1) = 0.56\\ P(x|C_2) = f_{\mu^2, \sum^2}(x)\\ P(C_2) = 0.44 P(C1​∣x)=P(x∣C1​)P(C1​)+P(x∣C2​)P(C2​)P(x∣C1​)P(C1​)​P(x∣C1​)=fμ1,∑1​(x)=(2πD/2)1​∣∑1∣1/21​exp{−21​(x−μ1)T(∑1)−1(x−μ1)}P(C1​)=0.56P(x∣C2​)=fμ2,∑2​(x)P(C2​)=0.44

4、Modyfying Model

使两个 class 有同样的方差 $\Sigma$.
$$L({\mu }^1,{\mu }^2,\Sigma )=f_{{\mu }^1,\Sigma }(x^1)f_{{\mu }^1,\Sigma }(x^2)\dots f_{{\mu }^1,\Sigma }(x^{7}9)\times f_{{\mu }^2,\Sigma }(x^{8}0)f_{{\mu }^2,\Sigma }(x^{8}1)\dots f_{{\mu }^2,\Sigma }(x^{1}40)$$

$$\begin{gathered} P(C_1|x)=\frac{P(x|C_1)P(C_1)}{P(x|C_1)P(C_1)+P(x|C_2)P(C_2)}=\frac{1}{1+\frac{P(x|C_2)P(C_2)}{P(x|C_1)P(C_1)}} \\ 令z=ln\frac{P(x|C_1)P(C_1)}{P(x|C_2)P(C_2)} \\ P(C_1|x)=\frac{1}{1+exp(-z)}=\sigma (z)(Sigmoidfunction)=\sigma (w\cdot x+b) \end{gathered}$$

---

二、Logistic Regression

1、Step 1: Function Set

2、Goodness of a Function

(1)、Cross Entropy

$$H(p,q)=-\sum _{x}p(x)ln(q(x))=\sum _n-[{\hat{y}}^{n}ln{f}_{w,b}(x^n)+(1-{\hat{y}}^n)ln(1-f_{w,b}(x^n))]$$

3、Find the best function

$$\frac{\partial lnL(w,b)}{\partial w_i}-lnL(w,b)=\frac{\partial ln{f}_{w,b}(x)}{\partial z}\frac{\partial z}{\partial w_i}$$

(1)、Cross Entropy vs. Squarre Error

4、Multi-class Classification

• Softmax:
  $$y_n=\frac{e^{z_n}}{{\sum }_{j=1}^n{e}^{z_j}}$$

---

三、Intro to Deep Learning

1、Step 1: Neural Network

2、Step 2: Goodness of function

• Total Loss:
  

3、Step 3: pick the best function (Gradient Descent)

---