【Machine Learning】Supervised Learning

本笔记基于清华大学《机器学习》的课程讲义监督学习相关部分，基本为笔者在考试前一两天所作的Cheat Sheet。内容较多，并不详细，主要作为复习和记忆的资料。

Linear Regression

Perceptron

• $f(x)=sign(w^{\top }x+b)$
• convergence

Logistic Regression

• output probability instead of labels.
• Loss: Cross entropy $XE(y,p)=-{\sum }_i{y}_i\log p_i$. $y_i$ is the actual probability

Ridge Regression

• $l_2$ regularization $\frac{\lambda }{2}\Vert w{\Vert }_2^2$
• Shrink every coordinate: $w^{\prime }=w\cdot (1-\eta \lambda )$
  • weight decay

LASSO Regression

• Find sparse features: Want $\Vert w{\Vert }_0\le c$
• $l_1$ regularization $\lambda \Vert w{\Vert }_1$
• Gradient: add or minus $\eta \lambda$ to pull $w$ into $0$.

Compressed Sensing*

• Compare with LASSO: pick $X_{train}$, observe $Y_{train}$, and learn $w$. Instead of given $X_{train},Y_{train}$.
• RIP condition: A matrix $A$ is $(ϵ,s)$-RIP if $\Vert Ax\Vert \approx \Vert x\Vert$ for $s$-sparse $x$.
• Application: reconstruct $x$ on a RIP matrix is easy.
  • For sparse $x$
  • For spase $x$ + some noise
  • Lemma: RIP implies almost orthogonality
  • Proof*
• How to find RIP matrix: Random
• Non-linear sparse: $X=G(Z)$, $Z$ is sparse, $G$ is non-linear.

SVM

• Minimize $\Vert w{\Vert }_2+\lambda {\sum }_i{\xi }_i$.
  • Soft version: $y_i{w}^{\top }{x}_i\ge 1-{\xi }_i,{\xi }_i\ge 0$.
  • maximize margin $1/\Vert w{\Vert }_2$.

Dual derivation

• Primal problem
  $$\begin{gathered} \underset{x}{\min }{x}^2 \\ x\ge b \end{gathered}$$

• Lagrangian $L(x,\alpha )=x^2-\alpha (x-b)\text{ s.t. }\alpha \ge 0$

  • Solution in primal problem correponds to a $L(x,\alpha )\le x^2$. Thus ${\min }_{x}L(x,\alpha )$ is lower bound of the primal solution.
  • Dual: We want to find the maximum lowerbound ${\max }_{\alpha }d(\alpha )={\max }_{\alpha }{\min }_{x}L(x,a)$.
  • Strong duality $p^{\ast }=d^{\ast }$

• For SVM(hard version)

  • Primal: $\min \frac{\Vert w{\Vert }_2}{2}$, $y_i{w}^{\top }{x}_i\ge 1$.

  • Dual: maximize(solved by taking derivative)
    $$\begin{gathered} L(w,\alpha )=\frac{\Vert w{\Vert }_2}{2}-\sum _i\alpha (y_i{w}^{\top }{x}_i-1) \\ =\sum _i{\alpha }_i-\frac{1}{2}\sum _i\sum _j{y}_i{y}_j{\alpha }_i{\alpha }_j⟨x_i,x_j⟩ \\ \alpha \ge 0 \\ w=\sum _i{\alpha }_i{y}_i{x}_i \end{gathered}$$

Kernel Method

• Replace $⟨x_i,x_j⟩$ with $⟨\varphi (x_i),\varphi (x_j)⟩$ to embed $x$ into a high dimension space.
• Use $k(x_i,x_j)$ to compute $\langle \phi(x_i),\phi(x_j)\rangle $. Usually gaussian kernel $e^{-\frac{|x_i-x_j|}{2{\sigma }^2}}$
• Mercer’s theorem: positive semidefinite kernel matrix has a corresponding embedding $\varphi$.

Decision Tree

Boolean funcional analysis

• ${\mathcal{X}}_S(x)={\prod }_{i\in S}{x}_i$

• $f(x)={\sum }_S\widehat{f_S}{\mathcal{X}}_S(x)$

• $\widehat{f_S}=⟨f,{\mathcal{X}}_S⟩={\mathbb{E}}_{x\sim D}[f(x){\mathcal{X}}_S(x)]$

• ${\mathbb{E}}_{x\sim D}[f(x{)}^2]={\sum }_S\widehat{f_S^2}$

• Decision tree with $s$ leaf nodes can be converted into $\log (\frac{s}{ϵ})$-degree, sparsity-$\frac{s^2}{ϵ}$ function that $2ϵ$-appoximates $T$.

  • log depth
  • Combination of AND term
  • Take the bases $S$ with big $\widehat{f_S}$

• Estimation

  • LMN: $\widehat{f_S}={\mathbb{E}}_{x\sim D}[f(x){\mathcal{X}}_S(x)]=\frac{1}{m}{\sum }_{i=1}^{m}f(x_i){\mathcal{X}}_S(x_i)$
    • Not work well in practice. Have not guarantees in noisy setting
  • Compressed Sensing
    • $y=Ax+e$: $x$ contains $\widehat{f_S}$, $A$ and $y$ are from samples.
    • Lasso find $x^{\ast }$ with bounded error.

Splitting variables

• Greedy: Gini index*

Random forest

• overfitting problem
• Construct $B$ trees, every tree is trained by $n$ samples, which is from $\{(x_i,y_i){\}}_{i=1}^n$ with replacement(可重复), each element will miss with probability $(1-\frac{1}{n}{)}^n=\frac{1}{e}$
• Output the average of $B$ trees
• Can also bag the features

Adaboost*

• Combine weak learners. Make hard cases more likely.

• Sample distribution $D_t(i)$ in $t$ round. ${ϵ}_t={\mathrm{Pr}}_{i\sim D_t}[h_t(x_i)\ne y_i]$.

• $D_1(i)=\frac{1}{m}$Learn $h_t$ from $D_t(i)$
  $$\begin{gathered} D_{t+1}(i)=\{\begin{array}{llc}\frac{1}{Z_t}{D}_t(i)e^{-{\alpha }_t}& & y_i=h_t(x_i)\\ \frac{1}{Z_t}{D}_t(i)e^{{\alpha }_t}& & y_i\ne h_t(x_i)\end{array} \\ \alpha =\frac{1}{2}\ln \left(\frac{1-{ϵ}_t}{{ϵ}_t}\right) \end{gathered}$$

• $H_{final}=sign(\sum {\alpha }_t{h}_t)$

• ${ϵ}_t=\frac{1}{2}-{\gamma }_t$.

  • ${\gamma }_t$ means how better this weak learner is than random classifier.

  $$error(H_{final})\le \prod _{t}2\sqrt{{ϵ}_t(1-{ϵ}_t)}\le \prod _t\sqrt{1-4{\gamma }_t^2}\le \exp \left(-2\sum _t{\gamma }_t^2\right)$$

• Proof:

  • $$D_T(i)=\frac{1}{m}\frac{\exp (-y_i{\sum }_t{\alpha }_t{h}_t(x_i)}{{\prod }_t{Z}_t}$$

  • $$\begin{array}{rl}error(H_{final})& =\frac{1}{m}\sum _{i=1}^m{1}_{y_{i}f(x_i)\le 0}\\ & =\frac{1}{m}\sum _{i=1}^m\exp (-y_{i}f(x_i))\\ & =\sum _{i=1}^m{D}_T(i)\prod _t{Z}_t\\ & =\prod _t{Z}_t\end{array}$$

  • $$Z_t=\sum _i{D}_t(i)\exp (-{\alpha }_t{y}_{i}h(x_i))=(1-{ϵ}_t)e^{-{\alpha }_t}+{ϵ}_t{e}^{{\alpha }_t}$$

    To minimize $Z_t$, ${\alpha }_t=\frac{1}{2}\ln (\frac{1-{ϵ}_t}{{ϵ}_t})$, then $Z_t=2\sqrt{{ϵ}_t(1-{ϵ}_t)}$

• Generalization

• Drawback: Only binary classification

• Extension: Gradient boosting: Regression, minimize $\frac{1}{2}{\sum }_i(F(x_i)-y_i{)}^2$

  • Adaboost use Coordinate descent: change ${\alpha }_t$ from $0$ to ${\alpha }_t$ in round $t$. Only change one coordinate since the dimension is too large.
  • learn a new regression tree $h(x)$ to fit $\partial L/\partial F(x_i)=F(x_i)-y$ for square loss. Then update $F^{\prime }(x)=F(x)+\eta h(x)$