Boosting

Boost

Gradient boosting

Idea

we can think our prediction function itself is a parameter $f(x)$. So the update of it is the gradient diction. For each i
$$f(x{)}_{t+1}=f(x{)}_t+\gamma \frac{\partial L(y,f(x))}{\partial f(x)}$$
As our model is additive of mutiple weak learner, so the next weak learner is suppose to best fit the gradient ter . The data set is $(x_i,\frac{\partial L(y,f(x{)}_i)}{\partial f(x{)}_i})$ to get a new predictor $h_m$

Find $\gamma$ such that

$${\gamma }_m=\underset{\gamma }{\mathrm{argmin}}\sum _{i=1}^{n}L\left(y_i,f_t(x_i)+\gamma h_m(x_i)\right).$$
One of the example is Gradient Boost Decision Tree.
link

Adaboosting

idea

for classification, our final predict function is still weight sum of weak learners.

Instead of using 0-1 loss, we use exponential loss functions,

The details are in wiki. Basically we adjuct our weigth on each data set, more weight will be put on the data point where we previously classify it wrongly.
Here is the link

link

Xboosting Tree vs GBDT

GBDT is using gradient boosting,

Xboosting Tree is using gradient boosting with second order information, basically it expand the Loss function with second order Taylor equations. our $h_m$ is considered as delta pertub in the Taylor expansion. Then find the optimal additive weak learner $h_m$ such that minimize this Taylor equations.

link