Summary of Boosted Tree and Factorization Machines

Boosted Tree

Definition:

$$\hat{y}=\sum _{k=1}^K{f}_k(x)$$

In which $f_k(x)$ is one of K regression trees.

Loss:

$$Loss=\sum _{i=1}^{n}L(y_i,{\hat{y}}_i)$$

Add some regularization:

$$Loss=\sum _{i=1}^{n}L(y_i,{\hat{y}}_i)+\sum _{k=1}^K\Omega (f_k)$$

Additive Training:

$${\hat{y}}^{(1)}=0$$

$${\hat{y}}^{(t)}={\hat{y}}^{(t-1)}+f_t(x)$$

$$Los{s}^{(t)}=\sum _{i=1}^{n}L(y_i,{\hat{y}}_i^{(t)})+\sum _{k=1}^t\Omega (f_k)$$

$$=\sum _{i=1}^{n}L(y_i,{\hat{y}}_i^{(t-1)}+f_t(x_i))+\sum _{k=1}^{t-1}\Omega (f_k)+\Omega (f_t)$$

$$=\sum _{i=1}^{n}L(y_i,{\hat{y}}_i^{(t-1)}+f_t(x_i))+\Omega (f_t)+C$$

$$\approx \sum _{i=1}^n[L(y_i,{\hat{y}}_i^{(t-1)})+f_t(x_i)\frac{\partial L}{\partial {\hat{y}}_i^{(t-1)}}+\frac{1}{2}{f}_t^2(x_i)\frac{\partial L^2}{\partial {\hat{y}}_i^{(t-1)}}]+\Omega (f_t)+C$$

$$=\sum _{i=1}^n[L(y_i,{\hat{y}}_i^{(t-1)})+f_t(x_i)G_i+\frac{1}{2}{f}_t^2(x_i)H_i]+\Omega (f_t)+C$$

$$=\sum _{i=1}^n[f_t(x_i)G_i+\frac{1}{2}{f}_t^2(x_i)H_i]+\Omega (f_t)+C^{\prime }$$

Loss at time t is:

$$Los{s}^{(t)}=\sum _{i=1}^n[f_t(x_i)G_i+\frac{1}{2}{f}_t^2(x_i)H_i]+\Omega (f_t)+C^{\prime }$$

Use:

f t ( x ) = w q ( x ) , q : R d → { 1 , 2 , . . . , M } , w i ∈ R f_t(x)=w_{q(x)}, q:R^d\rightarrow\{1,2,...,M\}, w_i \in R ft​(x)=wq(x)​,q:Rd→{1,2,...,M},wi​∈R

$$\Omega (f)=\frac{1}{2}\lambda \sum _{i=1}^M{w}_j^2+\gamma M$$

We get:

$$Los{s}^{(t)}=\sum _{i=1}^n[f_t(x_i)G_i+\frac{1}{2}{f}_t^2(x_i)H_i]+\Omega (f_t)+C^{\prime }$$

$$=\sum _{i=1}^n[w_{q(x_i)}{G}_i+\frac{1}{2}{w}_{q(x_i)}^2{H}_i]+\frac{1}{2}\lambda \sum _{j=1}^M{w}_j^2+\gamma M+C^{\prime }$$

With I j = { i ∣ q ( x i ) = j } I_j=\{i|q(x_i)=j\} Ij​={i∣q(xi​)=j}:

$$\sum _{i=1}^n{w}_{q(x_i)}{G}_i=\sum _{j=1}^M[w_j\sum _{i\in I_j}{G}_i]$$

$$\sum _{i=1}^n\frac{1}{2}{w}_{q(x_i)}^2{H}_i=\sum _{j=1}^M{w}_j^2\sum _{i\in I_j}\frac{1}{2}{H}_i$$

So:

$$Los{s}^{(t)}=\sum _{j=1}^M[w_j\sum _{i\in I_j}{G}_i+w_j^2\sum _{i\in I_j}\frac{1}{2}{H}_i+\frac{1}{2}\lambda w_j^2]+\gamma M+C^{\prime }$$

$$=\sum _{j=1}^M[w_j\sum _{i\in I_j}{G}_i+\frac{1}{2}{w}_j^2(\lambda +\sum _{i\in I_j}{H}_i)]+\gamma M+C^{\prime }$$

With $G_j^{\prime }={\sum }_{i\in I_j}{G}_i,H_j^{\prime }={\sum }_{i\in I_j}{H}_i$:

$$Los{s}^{(t)}=\sum _{j=1}^M[w_j{G}_j^{\prime }+\frac{1}{2}{w}_j^2(\lambda +H_j^{\prime })]+\gamma M+C^{\prime }$$

Finally:

$$w_j^{\ast }=argmin(w_j{G}_j^{\prime }+\frac{1}{2}{w}_j^2(\lambda +H_i^{\prime }))=-\frac{G_j^{\prime }}{\lambda +H_i^{\prime }}$$

$$Ob{j}^{(t)}=min(Los{s}^{(t)})=-\frac{1}{2}\sum _{j=1}^M\frac{G_j^{\prime 2}}{H_j^{\prime }+\lambda }+\gamma M+C^{\prime }$$

So for each iteration t of training, greedily seach for a regression tree $f_t(x_i)=w_{q(x_i)}$ with $w_j=-\frac{G_j^{\prime }}{\lambda +H_i^{\prime }}$ with minumum $Ob{j}^{(t)}$ and add it to model.

Factorization Machines

$$y=w_0+\sum _{i=1}^n{w}_i{x}_i+\sum _{i=1}^{n-1}\sum _{j=i+1}^n<{\mathbf{v}}_i,{\mathbf{v}}_j>x_i{x}_j$$

In which

$$w_0\in R,\mathbf{w}\in R^n,\mathbf{v}\in R^{n\times k}$$

$$<{\mathbf{v}}_i,{\mathbf{v}}_j>=\sum _{l=1}^k{v}_{il}{v}_{jl}$$