Loss Function 整理

最新推荐文章于 2021-11-10 21:41:57 发布

VeChu_LesChats

最新推荐文章于 2021-11-10 21:41:57 发布

阅读量438

点赞数

分类专栏： Machine Learning

本文链接：https://blog.csdn.net/vechu/article/details/72824339

版权

Machine Learning 专栏收录该内容

1 篇文章 0 订阅

订阅专栏

usepackage{txfonts}

机器学习常用Loss Function 整理

1. Squared Loss – Linear Regression

J (θ) = 1 2 m \sum i = 1 m (h θ (x (i)) - y (i)) 2

$J(\theta)=\frac{1}{2m}\sum_{i=1}^m(h_{\theta}(x^{(i)})-y^{(i)})^2$

2. 绝对值损失

J (θ) = 1 2 m \sum i = 1 m | h θ (x (i)) - y (i) |

$J(\theta)=\frac{1}{2m}\sum_{i=1}^m|h_{\theta}(x^{(i)})-y^{(i)}|$

3. log loss/cross entropy

J (θ) = - 1 m [\sum i = 1 m (y (i) l o g h θ (x (i)) + (1 - y (i)) l o g (1 - h θ (x (i))))]

$J(\theta)=-\frac{1}{m}\left[\sum_{i=1}^{m}(y^{(i)} log h_{\theta}(x^{(i)})+(1-y^{(i)})log (1-h_{\theta}(x^{(i)})))\right]$

4. Softmax loss function

J (θ) = - 1 m ⎡ ⎣ \sum i = 1 m \sum j = 1 k 1 {y (i) = j} l o g e θ T j x ( i ) \sum k l = 1 e θ T l x ( i ) ⎤ ⎦ = - 1 m ⎡ ⎣ \sum i = 1 m \sum j = 1 k 1 {y (i) = j} log p (y (i) = j | x (i); θ) ⎤ ⎦

$\begin{align} J(\theta) &= -\frac{1}{m}\left[\sum_{i=1}^{m}\sum_{j=1}^{k}1\{y^{(i)}=j\}\ log \frac{e^{\theta_j^Tx^{(i)}}}{\sum_{l=1}^ke^{\theta_l^Tx^{(i)}}}\right] \\ &= -\frac{1}{m}\left[\sum_{i=1}^{m}\sum_{j=1}^{k}1\{y^{(i)}=j\} \log \ p(y^{(i)}=j|x^{(i)};\theta)\right] \end{align}$

如果将下式中的 $\sum_{j=1}^{k}$ 替换成 $\sum_{j=0}^{1}$ ，即可得到 log loss

5. Hinge loss – SVM

二类分类问题:

J (w ⃗) = [1 m \sum i = 1 m m a x (0, 1 - y (i) (w ⃗ \cdot x (i) \to + b))] + λ ∥ w ⃗ ∥ 2

$\begin{equation} J(\vec w)=\left[\frac{1}{m}\sum_{i=1}^{m}max(0,1-y^{(i)}(\vec w \cdot\vec{x^{(i)}}+b))\right]+\lambda\|\vec w\|^2 \end{equation}$
或者可以写成：

J (w ⃗) = 1 m \sum i = 1 m ζ (i) + λ ∥ w ⃗ ∥ 2 (1) s u b j e c t t o y (i) ((w ⃗ \cdot x (i) \to + b) \geq 1 - ζ (i) a n d ζ (i) \geq 0, f o r a l l i

$J(\vec w)=\frac{1}{m}\sum_{i=1}^{m}\zeta^{(i)}+\lambda\|\vec w\|^2 \ \ \ \ (1) \\subject\ to\ y^{(i)}((\vec w \cdot\vec{x^{(i)}}+b)\ge1-\zeta^{(i)}\ and \ \zeta^{(i)}\ge0,\ for\ all\ i$

多类分类问题

J (w) = ⎡ ⎣ 1 m \sum i = 1 m \sum j \neq y (i) m a x (0, 1 + w T j \cdot x (i) \to - w T y (i) \cdot x (i) \to) ⎤ ⎦ + λ ∥ w ∥ 2

$J(\textbf w)=\left[\frac{1}{m}\sum_{i=1}^{m}\sum_{j\ne y^{(i)}} max(0,1+\textbf w_j^T \cdot\vec{x^{(i)}}-\textbf w_{y^{(i)}}^T \cdot\vec{x^{(i)}} )\right]+\lambda\|\textbf w\|^2$
或者可以写成：

J (w) = 1 m \sum i = 1 m ζ (i) + λ ∥ w ∥ 2 (2) s u b j e c t t o (i) ((w T y (i) \cdot x (i) \to - w T j \cdot x (i) \to) \geq e j i - ζ (i) i = 1, . . ., m, j = 1, . . ., k, y (i) \in {1, . . ., k}

$J(\textbf w)=\frac{1}{m}\sum_{i=1}^{m}\zeta^{(i)}+\lambda\|\textbf w\|^2 \ \ \ \ (2) \\subject\ to\ \ \ \ \begin{array} y^{(i)}((\textbf w_{y^{(i)}}^T \cdot\vec{x^{(i)}}-\textbf w_j^T \cdot\vec{x^{(i)}})\ge e_i^j-\zeta^{(i)}\ \\ i = 1, ..., m,\ j=1, ..., k,\ y^{(i)} \in \{1, ..., k\} \end{array}$
where

e j i = {01 if y (i) = j if y (i) \neq j

$e_i^j= \begin{cases} 0& \text{if $y^{(i)} =j$}\\ 1& \text{if $y^{(i)} \ne j$} \end{cases}$

6. L2 Hinge loss – SVM

将式(1), (2)改写成

J (w ⃗) = 1 m \sum i = 1 m ζ (i) 2 + λ ∥ w ⃗ ∥ 2 s u b j e c t t o y (i) ((w ⃗ \cdot x (i) \to + b) \geq 1 - ζ (i) a n d ζ (i) \geq 0, f o r a l l i

$J(\vec w)=\frac{1}{m}\sum_{i=1}^{m}{\zeta^{(i)}}^2+\lambda\|\vec w\|^2 \\subject\ to\ y^{(i)}((\vec w \cdot\vec{x^{(i)}}+b)\ge1-\zeta^{(i)}\ and \ \zeta^{(i)}\ge0,\ for\ all\ i$
以及

J (w) = 1 m \sum i = 1 m ζ (i) 2 + λ ∥ w ∥ 2 s u b j e c t t o (i) ((w T y (i) \cdot x (i) \to - w T j \cdot x (i) \to) \geq e j i - ζ (i) i = 1, . . ., m, j = 1, . . ., k, y (i) \in {1, . . ., k}

$J(\textbf w)=\frac{1}{m}\sum_{i=1}^{m}{\zeta^{(i)}}^2+\lambda\|\textbf w\|^2 \\subject\ to\ \ \ \ \begin{array} y^{(i)}((\textbf w_{y^{(i)}}^T \cdot\vec{x^{(i)}}-\textbf w_j^T \cdot\vec{x^{(i)}})\ge e_i^j-\zeta^{(i)}\ \\ i = 1, ..., m,\ j=1, ..., k,\ y^{(i)} \in \{1, ..., k\} \end{array}$