MLlib - Optimization Module - Gradient

最新推荐文章于 2024-10-04 17:33:46 发布

sjc314

最新推荐文章于 2024-10-04 17:33:46 发布

阅读量515

点赞数

本文链接：https://blog.csdn.net/sjc314/article/details/45096891

版权

machine-learning 同时被 3 个专栏收录

4 篇文章 0 订阅

订阅专栏

spark

4 篇文章 0 订阅

订阅专栏

MLlib module

4 篇文章 0 订阅

订阅专栏

MLlib - Optimization Module - Gradient

@(Hadoop & Spark)[machine learning|algorithm|statistics|Spark]

Topic: Gradient - LogisticGradient

Inference process

probility
$P (y = 0 | x, w) = 1 / (1 + \sum i K - 1 exp (x w i))$ $P(y=0|x, w) = 1 / (1 + \sum_i^{K-1} \exp(x w_i))$
$P (y = 1 | x, w) = e x p (x w 1) / (1 + \sum i K - 1 exp (x w i))$ $P(y=1|x, w) = exp(x w_1) / (1 + \sum_i^{K-1} \exp(x w_i))$
$. . .$ $...$
$P (y = K - 1 | x, w) = e x p (x w K - 1) / (1 + \sum i K - 1 exp (x w i))$ $P(y=K-1|x, w) = exp(x w_{K-1}) / (1 + \sum_i^{K-1} \exp(x w_i))$
loss function
$l (w, x) = - l o g P (y | x, w) = - α (y) l o g P (y = 0 | x, w) - (1 - α (y)) l o g P (y | x, w) = l o g (1 + \sum i K - 1 exp (x w i)) - (1 - α (y)) x w y - 1 = l o g (1 + \sum i K - 1 exp (m a r g i n s i)) - (1 - α (y)) m a r g i n s y - 1$ $l(w, x) = -log P(y|x, w) = -\alpha(y) log P(y=0|x, w) - (1-\alpha(y)) log P(y|x, w) \\ = log(1 + \sum_i^{K-1}\exp(x w_i)) - (1-\alpha(y)) x w_{y-1} \\ = log(1 + \sum_i^{K-1}\exp(margins_i)) - (1-\alpha(y)) margins_{y-1}$
where $α (i) = 1 i f i! = 0, a n d α (i) = 0 i f i = = 0, m a r g i n s i = x w i$ $\alpha(i) = 1 \ if \ i \ != 0, and \\ \alpha(i) = 0 \ if \ i == 0, \\ margins_i = x w_i$
first derivative
$\partial l ( w , x ) \partial w i j = (exp ( x w i ) ( 1 + \sum K - 1 k exp ( x w k ) ) - (1 - α (y) δ y, i + 1)) * x j = m u l t i p l i e r i * x j$ $\frac{\partial l(w, x)}{\partial w_{ij}} = (\frac{\exp(x w_i)} {(1 + \sum_k^{K-1} \exp(x w_k))} - (1-\alpha(y)\delta_{y, i+1}))* x_j \\ = multiplier_i * x_j$
where $δ i, j = 1 i f i = = j, δ i, j = 0 i f i! = j, a n d m u l t i p l i e r = exp ( m a r g i n s i ) ( 1 + \sum K - 1 k exp ( m a r g i n s i ) ) - (1 - α (y) δ y, i + 1)$ $\delta_{i, j} = 1 \ if \ i == j, \\ \delta_{i, j} = 0 \ if \ i != j, and \\ multiplier = \frac{\exp(margins_i)}{(1 + \sum_k^{K-1} \exp(margins_i))} - (1-\alpha(y)\delta_{y, i+1})$

Arithmetic overflow
when max(margins) > 0 Arithmetic overflow happen so the loss function and the multiplier need rewritten as below:
$l (w, x) = l o g (1 + \sum i K - 1 exp (m a r g i n s i)) - (1 - α (y)) m a r g i n s y - 1 = l o g (exp (- m a x M a r g i n) + \sum i K - 1 exp (m a r g i n s i - m a x M a r g i n)) + m a x M a r g i n - (1 - α (y)) m a r g i n s y - 1 = l o g (s u m + 1) + m a x M a r g i n - (1 - α (y)) m a r g i n s y - 1$ $l(w, x) = log(1 + \sum_i^{K-1}\exp(margins_i)) - (1-\alpha(y)) margins_{y-1} \\ = log(\exp(-maxMargin) + \sum_i^{K-1}\exp(margins_i - maxMargin)) + maxMargin \\ - (1-\alpha(y)) margins_{y-1} \\ = log(sum + 1) + maxMargin - (1-\alpha(y)) margins_{y-1}$
$m u l t i p l i e r = exp ( m a r g i n s i ) ( 1 + \sum K - 1 k exp ( m a r g i n s i ) ) - (1 - α (y) δ y, i + 1) = exp ( m a r g i n s i - m a x M a r g i n ) ( 1 + s u m ) - (1 - α (y) δ y, i + 1)$ $multiplier = \frac{\exp(margins_i) }{ (1 + \sum_k^{K-1} \exp(margins_i))} - (1-\alpha(y)\delta_{y, i+1}) \\ = \frac{\exp(margins_i - maxMargin) }{ (1 + sum) }- (1-\alpha(y)\delta_{y, i+1})$
where $s u m = exp (- m a x M a r g i n) + \sum i K - 1 exp (m a r g i n s i - m a x M a r g i n) - 1$ $sum = \exp(-maxMargin) + \sum_i^{K-1}\exp(margins_i - maxMargin) - 1$

reference

In The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition by Trevor Hastie, Robert Tibshirani, and Jerome Friedman (which can be downloaded from http://statweb.stanford.edu/~tibs/ElemStatLearn/ , Eq. (4.17) on page 119 gives the formula of multinomial logistic regression model)