Xgboost中对于二分类和多分类的Gradient和Hessian的求法

最新推荐文章于 2022-04-18 01:29:59 发布

遇见更好的自己

最新推荐文章于 2022-04-18 01:29:59 发布

阅读量1.5k

点赞数

分类专栏：机器学习

本文链接：https://blog.csdn.net/yc1203968305/article/details/78321393

版权

机器学习专栏收录该内容

42 篇文章 2 订阅

订阅专栏

logistic 损失求导

首先logistic损失定义为

L (y, p) = - (y l o g (p) + (1 - y) l o g (1 - p))

$L(y,p)=-(ylog(p)+(1-y)log(1-p))$
其中

p=11+e−F(x) $p=\frac{1}{1+e^{-F(x)}}$

\partial L \partial F ( x ) = - (y 1 p p (1 - p) + (1 - y) 1 1 - p p (1 - p)) = - (y (1 - p) + (1 - y) p) = p - y

$\begin{align} \frac{\partial L}{\partial F(x)}& = -(y \frac{1}{p}p(1-p)+(1-y)\frac{1}{1-p}p(1-p)) \\ & = -(y(1-p)+(1-y)p) \\ & = p-y \\ \end{align}$

hessian=p(1−p) $hessian = p(1-p)$

softmax损失求导

首先softmax损失定义为

L (y, p) = - \sum k = 1 K y k l o g p k

$L(y,p)=-\sum_{k=1}^Ky_klog{p_k}$
其中

pk=eFk(x)∑Ki=1eFi(x) $p_k=\frac{e^{F_k(x)}}{\sum_{i=1}^Ke^{F_i(x)}}$

\partial L \partial F m ( x ) = - \sum k = 1 K y k 1 p k \partial p k \partial F m ( x ) = - y m (1 - p m) - \sum k \neq m y k 1 p k (- p k p m) = - y m (1 - p m) + \sum k \neq m y k p m) = \sum k = 1 K y k p m - y m = p m - y m

$\begin{align} \frac{\partial L}{\partial F_m(x)}& = -\sum_{k=1}^Ky_k \frac{1}{p_k}\frac{\partial p_k}{\partial F_m(x)}\\ & = -y_m(1-p_m)-\sum_{k\neq m}y_k\frac{1}{p_k}(-p_kp_m)\\ & = -y_m(1-p_m)+\sum_{k\neq m}y_kp_m)\\ & =\sum_{k=1}^Ky_kp_m-y_m\\ & =p_m-y_m\\ \end{align}$

$hessian=p_m(1-p_m)$

解释一下 $-\sum_{k=1}^K\frac{\partial p_k}{\partial F_m(x)}$
当 $k=m$ 时

- \sum k = 1 K \partial p k \partial F m ( x ) = \partial \partial f m (e f m \sum j e f j) = e f ' m \sum j e f j - e f m ( \sum j e f j ) ' ( \sum j e f j ) 2 = p m - (p m) 2

$\begin{align} -\sum_{k=1}^K\frac{\partial p_k}{\partial F_m(x)}&=\frac{\partial}{\partial f_m}(\frac{e^{f_m}}{\sum_je^{f_j}})\\ &=\frac{e^{f_m^{'}}\sum_je^{f_j}-e^{f_m}(\sum_je^{f_j})^{'}}{(\sum_je^{f_j})^2}\\ &=p_m-(p_m)^2 \end{align}$

当 $k \neq m$ 时

- \sum k = 1 K \partial p k \partial F m ( x ) = - \sum k = 1 K \partial \partial f m (e f k \sum j e f j) = - \sum k = 1 K e f k (e f m ( \sum j e f j ) 2) = - \sum k = 1 K p k p m

$\begin{align} -\sum_{k=1}^K\frac{\partial p_k}{\partial F_m(x)}&=-\sum_{k=1}^K\frac{\partial}{\partial f_m}(\frac{e^{f_k}}{\sum_je^{f_j}})\\ &=-\sum_{k=1}^Ke^{f_k}(\frac{e^{f_m}}{(\sum_je^{f_j})^2})\\ &=-\sum_{k=1}^Kp_kp_m \end{align}$

遇见更好的自己

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
Xgboost中对于二分类和多分类的Gradient和Hessian的求法

logistic 损失求导首先logistic损失定义为 L(y,p)=−(ylog(p)+(1−y)log(1−p))L(y,p)=-(ylog(p)+(1-y)log(1-p)) 其中p=11+e−F(x)p=\frac{1}{1+e^{-F(x)}} ∂L∂F(x)=−(y1pp(1−p)+(1−y)11−pp(1−p))=−(y(1−p)+(1−y)p)=p−y\begin{align
复制链接

扫一扫

专栏目录