adaboost算法原理及实现

模型概述

Adaboost模型属于boost模型中的一种，boost模型的思想是通过从弱学习算法出发，反复学习，得到一系列弱分类器（又称为基本分类器），然后组合这些弱分类器，得到相应的强分类器。大多数的boost方法都是改变训练数据的概率分布，然后针对不同的训练数据分布学习相应的弱分类器。

Adaboost的模型的思想是在每一次训练过程中提高被前一轮弱分类器的错误分类的样本的权重，这样可以让分类器更好的纠正错误。在训练完所有的分类器后，Adaboost采用的是加权多数表决的方法来进行投票，加大分类误差率小的分类器的权重，使其在表决中能够其较大作用。

Adaboost模型可以看成是加法模型的特例,形式如下：
$$f(x)=\sum _{m=1}^M{\alpha }_m{G}_m(x)$$
$G_m(x)$代表基分类器，${\alpha }_m$代表其系数

模型策略

Adaboost模型可以看成是加法模型，相应的损失函数可以是指数损失函数。
$$L(y,f(x))=exp[-yf(x)]$$
记$f_k(x)$为经过前k次学习后前k个弱学习器组合后的学习器。假设第k次迭代的参数${\alpha }_k,G_k(x)$,则$$f_k(x)=f_{k-1}(x)+{\alpha }_k{G}_k(x)$$
将上式代入损失函数可得：
$$loss=\sum _{i=1}^{N}exp[-y_i(f_{k-1}(x_i)+\alpha G(x_i))]$$
根据经验损失最小化的原则，有${\alpha }_k,G_k(x)$为：
$({\alpha }_k,G_k(x))=\underset{\alpha ,G(x)}{\mathrm{argmin}}\sum _{i=1}^{N}exp[-y_i(f_{k-1}(x)+\alpha G(x_i))]$

固定$\alpha$,则有使上式最小的$G_m^{\ast }(x)$应该是
$$G_m^{\ast }(x)=\underset{G}{\mathrm{argmin}}\sum _{i=1}^N{w}_{mi}^{-}I(y_i\ne G(x_i))$$
其中$w_{mi}^{-}=exp[-y_i{f}_{m-1}(x_i)]$

而对于${\alpha }_m^{\ast }$,从损失函数有：
$$\begin{gathered} \sum _{i=1}^N{w}_{mi}^{-}exp[-y_i\alpha G(x_i))]=\sum _{y_i=G(x_i)}{w}_{mi}^{-}{e}^{-\alpha }+\sum _{y_i\ne G(x_i)}{w}_{mi}^{-}{e}^{\alpha }= \\ (e^{\alpha }-e^{-\alpha })G\sum _{i=1}^N{w}_{mi}^{-}I(y_i\ne G(x_i))+e^{-\alpha }\sum _{i=1}^N{w}_{mi}^{-} \end{gathered}$$

对上式$\alpha$求导，则有
$${\alpha }_m^{\ast }=\frac{1}{2}log\frac{1-e_m}{e_m}$$
其中:$$e_m=\frac{\sum _{i=1}^N{w}_{mi}^{-}I(y_i\ne G(x_i))}{\sum _{i=1}^N{w}_{mi}^{-}}=\sum _{i=1}^N{w}_{mi}I(y_i\ne G(x_i))$$
最后每一轮权值的更新由：$$w_{mi}^{-}=exp[-y_i{f}_{m-1}(x_i)]$$,以及$$f_m(x)=f_{m-1}(x)+{\alpha }_k{G}_m(x)$$
可得：
$$w_{m+1,i}^{-}=w_{m,i}^{-}exp[-y_i{\alpha }_m{G}_m(x)]$$

模型算法

输入：训练数据集$T=(x_1,y_1),(x_2,y_2),...,(x_N,y_N))$,其中 x i ∈ X ⊆ R n , y i ∈ Y = { − 1 , + 1 } x_i\in X\subseteq R^n,y_i \in Y = \{-1,+1\} xi​∈X⊆Rn,yi​∈Y={−1,+1}
输出： 最终分类器$G(x)$
(1)初始化训练数据的权值分布
$$D_1=(w_{11},...,w_{1i},...,w_{1N}),w_{1i}=\frac{1}{N},i=1,2,...,N$$
(2)对m = 1,2,…,M
( a )使用具有权值分布$D_m$的训练数据集学习，得到基本分类器
G m ( x ) : X − > { − 1 , + 1 } G_m(x): X ->\{-1,+1\} Gm​(x):X−>{−1,+1}
( b )计算G_m(x)在训练数据集上的分类误差率
$$e_m=\frac{\sum _{i=1}^N{w}_{mi}^{-}I(y_i\ne G(x_i))}{\sum _{i=1}^N{w}_{mi}^{-}}=\sum _{i=1}^N{w}_{mi}I(y_i\ne G(x_i))$$
( c )计算$G_m$的系数
$${\alpha }_m=\frac{1}{2}log\frac{1-e_m}{e_m}$$
( d )更新训练数据集的权值分布
$$\begin{gathered} D_{m+1}=(w_{m+1,1},...,w_{m+1,i},...,w_{m+1,N}) \\ {w}_{m+1,i}=\frac{w_{m,i}}{Z_m}exp(-{\alpha }_m{y}_i{G}_m(x_i)) \\ {Z}_m=\sum _{i=1}^N{w}_{m,i}exp(-{\alpha }_m{y}_i{G}_m(x_i)) \end{gathered}$$
(3)生成最终分类器
$$G(x)=sign(\sum _{m=1}^M{\alpha }_m{G}_m(x))$$

代码实现

首先导入相关包

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import pandas as pd
import matplotlib.pyplot as pyplot

引入测试数据

def create_data():
    iris=load_iris()
    df=pd.DataFrame(iris.data,columns=iris.feature_names)
    df['label']=iris.target
    df.columns=['sepal length','sepal width','pedal_length','pedal width','label']
    data=df.iloc[:100,[0,1,-1]]
    data['label'].apply(lambda x: 1 if x==1 else -1)
    data = np.array(data)
    return data[:,:2] ,data[:,-1]

算法核心原理部分主要包括生成G(x)和$\alpha$的部分

class Adaboost:
	def __init__(self, n_estimators, learning_rate):
		self.n_estimators = n_estimators
		self.learning_rate = learning_rate
		self.model = []


	def fit(self,X_train, y_train):
		"""拟合训练集数据"""
		self.m, self.n = X_train.shape
		#初始化权值分布
		self.weight = [np.ones(self.m) / self.m]
		for i in range(self.n_estimators):
			compare_array, position, threshold, error, axis = self._G(X_train, y_train,self.weight[i])
			alpha_i = self.caculate_alpha(error)
			Z_i = self.caculate_Z(alpha_i, self.weight[i], compare_array, y_train)
	
			self.weight.append(self.weight[i] * np.exp(- alpha_i * compare_array * y_train /Z_i))

			self.model.append((axis, alpha_i, position, threshold))







	def caculate_alpha(self,error):
		return 0.5 * np.log((1 -error) / error)



	def caculate_Z(self, alpha, weight, pre_y, y):
		return np.dot(weight,np.exp(- alpha * pre_y *y))



	def calculate_err_rate(self, pre_y, y, weight):
		error = sum([weight[i] if pre_y[i] != y[i] else 0 for i in range(self.m)])  
		return error


	def G(self, threshold, x, position):
		#基本分类器
		if position == 'positive':
			pre_y = np.array([1 if x[i] >threshold else -1 for i in range(len(x))])
		else:
			pre_y = np.array([-1 if x[i] >threshold else 1 for i in range(len(x))])

		return pre_y

		
	def _G(self, X_train, y_train,weight):
		min_error = np.inf
		position = None
		threshold = None
		compare_array = None
		axis = None
		for i in range(self.n):
			feature = X_train[:, i]
			feature_max = max(feature)
			feature_min = min(feature)
			iter_num = int((feature_max -feature_min) // self.learning_rate)
    

			for j in range(iter_num):
				vi = feature_min + j * self.learning_rate

				pre_y_positive = self.G(vi, feature, 'positive')
				err_positive = self.calculate_err_rate(pre_y_positive, y_train,weight)
		
				pre_y_negative = self.G(vi, feature, 'negative')
				err_negative = self.calculate_err_rate(pre_y_negative, y_train, weight)

				if err_positive >err_negative:
					if err_negative < min_error:
						max_error = err_negative
						position = 'nagetive'
						compare_array = pre_y_negative
						threshold = vi
						axis = i
				else:
					if err_positive < min_error:
						max_error = err_positive
						position = 'positive'
						compare_array = pre_y_positive
						threshold = vi
						axis = i


		return compare_array, position, threshold, max_error, axis



	def predict(self, X_test, y_test):
		"""预测测试集数据"""
		result = []
		for i in range(len(self.model)):
			axis, alpha_i, position, threshold = self.model[i]
			result += alpha_i * self.G(threshold, X_test[i], y_test)

		return [1 if result[i] > 0 else -1 for i in range(len(result))]




	def score(self, X_test, y_test):
		"""测试模型正确率"""
		num = X_test.shape[0]
		acc_num = 0
		f = self.predict(X_test)
		acc_num = sum(f == y_test)
		return float(acc_num / num)