GBDT、XGBoost、LightGBM算法公式推导

一、GBDT公式推导

1、第一个基函数：
$$F_0(X)=\frac{1}{2}log\frac{1+\overline{y}}{1-\overline{y}}\text{\hspace{1em}(1.1)}$$
对于损失函数 L ( y , F ) = l o g ( 1 + e − 2 y F ) , y ∈ { − 1 , 1 } L(y, F)=log(1+e^{-2yF}), y\in \left\{-1, 1\right\} L(y,F)=log(1+e−2yF),y∈{ −1,1}，求损失函数最小对应的$F$值。求一阶导数：
$$L^{{}^{\prime }}=\frac{-2y{e}^{-2yF}}{1+e^{-2yF}}\text{\hspace{1em}(1.2)}$$
设
$$\{\begin{array}{ll}& count(y=+1)=m1\\ & count(y=-1)=m2\end{array}\text{\hspace{1em}(1.3)}$$
则有
$$\{\begin{array}{l}m1+m2=n\\ \frac{m1-m2}{n}=\overline{y}\end{array}\text{\hspace{1em}(1.4)}$$
即
$$\{\begin{array}{l}m1=\frac{n}{2}(1+\overline{y})\\ m2=\frac{n}{2}(1-\overline{y})\end{array}\text{\hspace{1em}(1.5)}$$
令$\sum L^{{}^{\prime }}=0$得到,
$$\sum L^{{}^{\prime }}=\sum _{y=1}{L}^{{}^{\prime }}+\sum _{y=-1}{L}^{{}^{\prime }}=0\text{\hspace{1em}(1.6)}$$
将$(1.5)$带入$(1.6)$，得到，
L ′ = ∑ y = 1 L ′ + ∑ y = − 1 L ′ = n 2 ( 1 + y ‾ ) ∗ − 2 e − 2 F 1 + e − 2 F + n 2 ( 1 − y ‾ ) ∗ 2 e 2 F 1 + e 2 F = n 2 ( 1 + y ‾ ) ∗ − 2 1 + e 2 F + n 2 ( 1 − y ‾ ) ∗ 2 e 2 F 1 + e 2 F = n 1 + e 2 F [ − ( 1 + y ‾ ) + e 2 F ( 1 − y ‾ ) ] (1.7) \begin{aligned} L^{'}&=\sum_{y=1}L^{'}+\sum_{y=-1}L^{'}\\\\ &=\frac{n}{2}(1+\overline{y})*\frac{-2e^{-2F}}{1+e^{-2F}} + \frac{n}{2}(1-\overline{y})*\frac{2e^{2F}}{1+e^{2F}}\\\\ &=\frac{n}{2}(1+\overline{y})*\frac{-2}{1+e^{2F}} + \frac{n}{2}(1-\overline{y})*\frac{2e^{2F}}{1+e^{2F}}\\\\ &=\frac{n}{1+e^{2F}}[-(1+\overline{y}) + e^{2F}(1-\overline{y})] \end{aligned} \tag{1.7} L′​=y=1∑​L′+y=−1∑​L′=2n​(1+y​)∗1+e−2F−2e−2F​+2n​(1−y​)∗1+e2F2e2F​=2n​(1+y​)∗1+e2F−2​+2n​(1−y​)∗1+e2F2e2F​=1+e2Fn​[−(1+y​)+e2F<