gbdt基本原理

为什么要建立多棵树

函数空间上的梯度下降

设样本为$\begin{array}{r}(x^j,y^j),j=1,2,\cdots ,n\end{array}$
对于回归问题，损失函数为
$$\begin{array}{rcl}L& =& {\sum }_{j=1}^{n}l(x^j,y^j,f^j)\\ & =& L(f^1,f^2,\cdots ,f^j)\\ & =& L(F)\end{array}$$
对于二分类问题，损失函数为
$$\begin{array}{rcl}L& =& {\sum }_{j=1}^{n}l(x^j,y^j,\sigma (f^j))\\ & =& L(f^1,f^2,\cdots ,f^j)\\ & =& L(F)\end{array}$$
F是多维函数$(f^1,f^2,\cdots ,f^j)$
目的要求
$$F=\underset{F}{\mathrm{argmin}}L(F)$$

根据梯度下降有
$$\begin{gathered} F_0=F_0 \\ {F}_1=F_0-\eta \nabla L{|}_{F=F_0} \\ \cdots \\ {F}_i=F_{i-1}-\eta \nabla L{|}_{F=F_{i-1}} \end{gathered}$$
所以有
$$F=F_0+\eta \sum _{i=0}^m-\nabla L{|}_{F=F_i}$$
取每个维度，
$$\begin{array}{rcl}{f}^j& =& f_0^j+\eta {\sum }_{i=0}^m-\frac{\partial L}{\partial f^j}{|}_{f^j=f_i^j}\\ & =& f_0^j+\eta {\sum }_{i=0}^m-\frac{\partial l}{\partial f^j}{|}_{f^j=f_i^j}\\ j& =& 1,2,\cdots ,n\end{array}$$
把$f^j$看成函数$f$，即
$$\begin{array}{rcl}f& =& \{\begin{array}{c}\begin{array}{c}\begin{array}{c}{f}^1,x=x^1\\ f^2,x=x^2\end{array}\end{array}\\ \cdots \\ f^n,x=x^n\end{array}=f(x)\end{array}$$
所以
$$f=f_0+\eta \sum _{i=0}^m-\frac{\partial l}{\partial f}{|}_{f=f_i}$$
令
$$\begin{gathered} T_0=f_0 \\ {T}_1=-\frac{\partial l}{\partial f}{|}_{f=f_0} \\ \cdots \\ {T}_i=-\frac{\partial l}{\partial f}{|}_{f=fi} \end{gathered}$$
则
$$f=T_0+\eta T_1+\cdots +\eta T_i$$
$T_0$为初始函数，可以随意定。
对于基分类器$T_i$，则有
$$T_i(x)=\{\begin{array}{c}\begin{array}{c}\begin{array}{c}-\frac{\partial l}{\partial f}{|}_{f=f_i^1},x=x^1\\ -\frac{\partial l}{\partial f}{|}_{f=f_i^2},x=x^2\end{array}\end{array}\\ \cdots \\ -\frac{\partial l}{\partial f}{|}_{f=f_i^n},x=x^n\end{array}$$

残差推导

对于回归问题，取平方损失$l=(y^j-f{)}^2$，则有
$$T_i=-\frac{\partial l}{\partial f}{|}_{f=f_i}=2(f_i-y^j)$$
对于二分类问题，取交叉熵损失$l=y^{j}ln(\sigma (f))+(1-y^j)ln(1-\sigma (f))$，则有
$$T_i=-\frac{\partial l}{\partial f}{|}_{f=f_i}=\sigma (f_i)-y^j$$

建树过程

第一棵树

第一棵树只有一个根节点，即对所有的$x^j$，都有唯一的输出c。
对于回归问题
$$\begin{gathered} T_0=c=\underset{c}{\mathrm{argmin}}(L(c)) \\ =\underset{c}{\mathrm{argmin}}(\sum _{j=1}^{n}l(x^j,y^j,c)) \\ =\underset{c}{\mathrm{argmin}}(\sum _{j=1}^n{(c-y^j)}^2) \end{gathered}$$
求导得到
$$\frac{\partial L}{\partial c}=2\sum _{j=1}^{n}c-y^j=0$$
解得
$$c=\frac{1}{n}\sum _{j=1}^n{y}^j$$
对于二分类问题
$$\begin{gathered} T_0=c=\underset{c}{\mathrm{argmin}}(L(c)) \\ =\underset{c}{\mathrm{argmin}}(\sum _{j=1}^{n}l(x^j,y^j,\sigma (c))) \\ =\underset{c}{\mathrm{argmin}}(\sum _{j=1}^n{y}^{j}ln(\sigma (c))+(1-y^j)ln(1-\sigma (c))) \end{gathered}$$
求导得到
$$\frac{\partial L}{\partial c}=\sum _{j=1}^n(\sigma (c)-y^j)=0$$
解得
$$c={\sigma }^{-1}(\frac{1}{n}\sum _{j=1}^n{y}^j)$$

第i棵树

如何分裂

这里选择CART回归树，对特征排序，遍历分裂点，把样本分为两个组，L和R
$$G=min(\sum _{y^j\in L}{(y^j-c_1)}^2+\sum _{y^j\in R}{(y^j-c_2)}^2)$$
解得
$$\begin{gathered} c_1=\frac{1}{L}\sum _{y^j\in L}{y}^j \\ {c}_2=\frac{1}{R}\sum _{y^j\in R}{y}^j \end{gathered}$$
取能够使G达到最小的特征及其对应的分裂点

如何取值

对于回归问题
$$\begin{gathered} c=\underset{c}{\mathrm{argmin}}(\sum _{y^j\in R}l(x^j,y^j,f_i^j+c)) \\ =\underset{c}{\mathrm{argmin}}{(\sum _{y^j\in R}(f_i^j+c-y^j)}^2) \end{gathered}$$
解得
$$c=\frac{1}{R}\sum _{y^j\in R}(y^j-f_i^j)$$
对于分类问题
$$\begin{gathered} c=\underset{c}{\mathrm{argmin}}(\sum _{y^j\in R}l(x^j,y^j,\sigma (f_i^j+c))) \\ =\underset{c}{\mathrm{argmin}}(\sum _{y^j\in R}{y}^{j}ln(\sigma (f_i^j+c))+(1-y^j)ln(1-\sigma (f_i^j+c))) \end{gathered}$$
令导数为0
$$\frac{\partial L}{\partial c}=\sum _{j\in R}(y^j-\sigma (f_i^j+c))=0$$
该函数的导数为
$$\frac{{\partial }^{2}L}{\partial c^2}=-\sum _{j\in R}\sigma (f_i^j+c)(1-\sigma (f_i^j+c))<0$$
函数递减，图像如下

利用牛顿一阶近似求零点，左边直线方程为
$$y=kx+b$$
$$\begin{gathered} k=\frac{{\partial }^{2}L}{\partial c^2}{|}_{c=0}=-\sum _{j\in R}\sigma (f_i^j)(1-\sigma (f_i^j)) \\ b=\frac{\partial L}{\partial c}{|}_{c=0}=\sum _{j\in R}{y}^j-\sigma (f_i^j) \end{gathered}$$
所以左边比较靠近c的红色圆圈的点的坐标为
$$\hat{c}=-\frac{b}{k}=\frac{{\sum }_{j\in R}{y}^j-\sigma (f_i^j)}{{\sum }_{j\in R}\sigma (f_i^j)(1-\sigma (f_i^j))}$$

多分类问题

假如有K>=3种分类，那么gbdt会创建K串树，每一串会在各自的梯度上分裂增长。但是梯度计算是相互依赖的。
$$L=\sum _{j=1}^{n}l(x^j,y^j,s(f^j))=\sum _{j=1}^n\sum _{k=1}^K{y}^{jk}\ln \frac{e^{f_i^{jk}}}{{\sum }_{t=1}^K{e}^{f_i^{jt}}}$$
如果$y^{jk}=1$，则
$$\frac{\partial L}{\partial f_i^{jk}}=\frac{{\sum }_{t=1}^K{e}^{f_i^{jt}}}{e^{f_i^{jk}}}\frac{e^{f_i^{jk}}{\sum }_{t=1}^K{e}^{f_i^{jt}}-{(e^{f_i^{jk}})}^2}{{({\sum }_{t=1}^K{e}^{f_i^{jt}})}^2}=1-\frac{e^{f_i^{jk}}}{{\sum }_{t=1}^K{e}^{f_i^{jt}}}$$
如果$y^{jk}=0$，则
$$\frac{\partial L}{\partial f_i^{jk}}=-\frac{e^{f_i^{jk}}}{{\sum }_{t=1}^K{e}^{f_i^{jt}}}$$
所以
$$\frac{\partial L}{\partial f_i^{jk}}=y^{jk}-\frac{e^{f_i^{jk}}}{{\sum }_{t=1}^K{e}^{f_i^{jt}}}$$
公式编辑网址
http://www.wiris.com/editor/demo/en/developers