3.1
式3.2 $f(x)=\omega ^{T}x+b$ 中,$\omega ^{T}$ 和b有各自的意义,简单来说,$\omega ^{T}$ 决定学习得到模型(直线、平面)的方向,而b则决定截距,当学习得到的模型恰好经过原点时,可以不考虑偏置项b。偏置项b实质上就是体现拟合模型整体上的浮动,可以看做是其它变量留下的偏差的线性修正,因此一般情况下是需要考虑偏置项的。但如果对数据集进行了归一化处理,即对目标变量减去均值向量,此时就不需要考虑偏置项了。
3.2
对区间[a,b]上定义的函数f(x),若它对区间中任意两点x1,x2均有$f(\frac{x1+x2}{2})\leq \frac{f(x1)+f(x2)}{2}$,则称f(x)为区间[a,b]上的凸函数。对于实数集上的函数,可通过二阶导数来判断:若二阶导数在区间上非负,则称为凸函数,在区间上恒大于零,则称为严格凸函数。
对于式3.18 $y=\frac{1}{1+e^{-(\omega ^{T}x+b)}}$,有
$\frac{dy}{d\omega ^{T}}=\frac{1}{(1+e^{-(\omega ^{T}x+b)})^{2}}e^{-(\omega ^{T}x+b)}(-x)=(-x)\frac{1}{1+e^{-(\omega ^{T}x+b)}}(1-\frac{1}{1+e^{-(\omega ^{T}x+b)}})=xy(y-1)=x(y^{2}-y)$
$\frac{d}{d\omega ^{T}}(\frac{dy}{d\omega ^{T}})=x(2y-1)(\frac{dy}{d\omega ^{T}})=x^{2}y(2y-1)(y-1)$
其中,y的取值范围是(0,1),不难看出二阶导有正有负,所以该函数非凸。
3.3
对率回归即Logis regression
西瓜集数据如图所示:
将好瓜这一列变量用0/1变量代替,进行对率回归学习,python代码如下:
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
dataset = pd.read_csv('/home/zwt/Desktop/watermelon3a.csv')
#数据预处理
X = dataset[['密度','含糖率']]
Y = dataset['好瓜']
good_melon = dataset[dataset['好瓜'] == 1]
bad_melon = dataset[dataset['好瓜'] == 0]
#画图
f1 = plt.figure(1)
plt.title('watermelon_3a')
plt.xlabel('density')
plt.ylabel('radio_sugar')
plt.xlim(0,1)
plt.ylim(0,1)
plt.scatter(bad_melon['密度'],bad_melon['含糖率'],marker='o',color='r',s=100,label='bad')
plt.scatter(good_melon['密度'],good_melon['含糖率'],marker='o',color='g',s=100,label='good')
plt.legend(loc='upper right')
#分割训练集和验证集
X_train,X_test,Y_train,Y_test = model_selection.train_test_split(X,Y,test_size=0.5,random_state=0)
#训练
log_model = LogisticRegression()
log_model.fit(X_train,Y_train)
#验证
Y_pred = log_model.predict(X_test)
#汇总
print(metrics.confusion_matrix(Y_test, Y_pred))
print(metrics.classification_report(Y_test, Y_pred, target_names=['Bad','Good']))
print(log_model.coef_)
theta1, theta2 = log_model.coef_[0][0], log_model.coef_[0][1]
X_pred = np.linspace(0,1,100)
line_pred = theta1 + theta2 * X_pred
plt.plot(X_pred, line_pred)
plt.show()
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
dataset = pd.read_csv('/home/zwt/Desktop/watermelon3a.csv')
#数据预处理
X = dataset[['密度','含糖率']]
Y = dataset['好瓜']
good_melon = dataset[dataset['好瓜'] == 1]
bad_melon = dataset[dataset['好瓜'] == 0]
#画图
f1 = plt.figure(1)
plt.title('watermelon_3a')
plt.xlabel('density')
plt.ylabel('radio_sugar')
plt.xlim(0,1)
plt.ylim(0,1)
plt.scatter(bad_melon['密度'],bad_melon['含糖率'],marker='o',color='r',s=100,label='bad')
plt.scatter(good_melon['密度'],good_melon['含糖率'],marker='o',color='g',s=100,label='good')
plt.legend(loc='upper right')
#分割训练集和验证集
X_train,X_test,Y_train,Y_test = model_selection.train_test_split(X,Y,test_size=0.5,random_state=0)
#训练
log_model = LogisticRegression()
log_model.fit(X_train,Y_train)
#验证
Y_pred = log_model.predict(X_test)
#汇总
print(metrics.confusion_matrix(Y_test, Y_pred))
print(metrics.classification_report(Y_test, Y_pred))
print(log_model.coef_)
theta1, theta2 = log_model.coef_[0][0], log_model.coef_[0][1]
X_pred = np.linspace(0,1,100)
line_pred = theta1 + theta2 * X_pred
plt.plot(X_pred, line_pred)
plt.show()
![](https://i-blog.csdnimg.cn/blog_migrate/e88f9ae1424f066e18ea088349a3fabc.png)
模型效果输出(查准率、查全率、预测效果评分):
precision recall f1-score support
Bad 0.75 0.60 0.67 5
Good 0.60 0.75 0.67 4
micro avg 0.67 0.67 0.67 9
macro avg 0.68 0.68 0.67 9
weighted avg 0.68 0.67 0.67 9
也可以输出验证集的实际结果和预测结果:
密度 含糖率 Y_test Y_pred
1 0.774 0.376 1 1
6 0.481 0.149 1 0
8 0.666 0.091 0 0
9 0.243 0.267 0 1
13 0.657 0.198 0 0
4 0.556 0.215 1 1
2 0.634 0.264 1 1
14 0.360 0.370 0 1
10 0.245 0.057 0 0
3.4
首先附上使用葡萄酒品质数据做的对率回归学习代码
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
import numpy as np import matplotlib.pyplot as plt import pandas as pd pd.set_option('display.max_rows',None) pd.set_option('max_colwidth',200) pd.set_option('expand_frame_repr', False) from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn import metrics dataset = pd.read_csv('/home/zwt/Desktop/winequality-red_new.csv') #数据预处理 dataset['quality2'] = dataset['quality'].apply(lambda x: 0 if x < 5 else 1) #新加入二分类变量是否为好酒,基于原数据中quality的值,其大于等于5就定义为好酒,反之坏酒 X = dataset[["fixed_acidity","volatile_acidity","citric_acid","residual_sugar","chlorides","free_sulfur_dioxide","total_sulfur_dioxide","density","pH","sulphates","alcohol"]] Y = dataset["quality2"] #分割训练集和验证集 X_train,X_test,Y_train,Y_test = model_selection.train_test_split(X,Y,test_size=0.5,random_state=0) #训练 log_model = LogisticRegression() log_model.fit(X_train,Y_train) #验证 Y_pred = log_model.predict(X_test) #汇总 print(metrics.confusion_matrix(Y_test, Y_pred)) print(metrics.classification_report(Y_test, Y_pred)) print(log_model.coef_)
其中,从UCI下载的数据集格式有问题,无法直接使用,先编写程序将格式调整完毕再使用数据
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
fr = open('/home/zwt/Desktop/winequality-red.csv','r',encoding='utf-8')
fw = open('/home/zwt/Desktop/winequality-red_new.csv','w',encoding='utf-8')
f = fr.readlines()
for line in f:
line = line.replace(';',',')
fw.write(line)
fr.close()
fw.close()
两种方法的错误率比较
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
from sklearn.linear_model import LogisticRegression from sklearn import model_selection from sklearn.datasets import load_wine # 载入wine数据 dataset = load_wine() #10次10折交叉验证法生成训练集和测试集 def tenfolds(): k = 0 truth = 0 while k < 10: kf = model_selection.KFold(n_splits=10, random_state=None, shuffle=True) for x_train_index, x_test_index in kf.split(dataset.data): x_train = dataset.data[x_train_index] y_train = dataset.target[x_train_index] x_test = dataset.data[x_test_index] y_test = dataset.target[x_test_index] # 用对率回归进行训练,拟合数据 log_model = LogisticRegression() log_model.fit(x_train, y_train) # 用训练好的模型预测 y_pred = log_model.predict(x_test) for i in range(len(x_test)): #这里和留一法不同,是因为10折交叉验证的验证集是len(dataset.target)/10,验证集的预测集也是,都是一个列表,是一串数字,而留一法是一个数字 if y_pred[i] == y_test[i]: truth += 1 k += 1 # 计算精度 accuracy = truth/(len(x_train)+len(x_test)) #accuracy = truth/len(dataset.target) print("用10次10折交叉验证对率回归的精度是:", accuracy) tenfolds() #留一法 def leaveone(): loo = model_selection.LeaveOneOut() i = 0 true = 0 while i < len(dataset.target): for x_train_index, x_test_index in loo.split(dataset.data): x_train = dataset.data[x_train_index] y_train = dataset.target[x_train_index] x_test = dataset.data[x_test_index] y_test = dataset.target[x_test_index] # 用对率回归进行训练,拟合数据 log_model = LogisticRegression() log_model.fit(x_train, y_train) # 用训练好的模型预测 y_pred = log_model.predict(x_test) if y_pred == y_test: true += 1 i += 1 # 计算精度 accuracy = true / len(dataset.target) print("用留一法验证对率回归的精度是:", accuracy) leaveone()
3.5
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn import model_selection from sklearn import metrics dataset = pd.read_csv('/home/zwt/Desktop/watermelon3a.csv') #数据预处理 X = dataset[['密度','含糖率']] Y = dataset['好瓜'] #分割训练集和验证集 X_train,X_test,Y_train,Y_test = model_selection.train_test_split(X,Y,test_size=0.5,random_state=0) #训练 LDA_model = LinearDiscriminantAnalysis() LDA_model.fit(X_train,Y_train) #验证 Y_pred = LDA_model.predict(X_test) #汇总 print(metrics.confusion_matrix(Y_test, Y_pred)) print(metrics.classification_report(Y_test, Y_pred, target_names=['Bad','Good'])) print(LDA_model.coef_) #画图 good_melon = dataset[dataset['好瓜'] == 1] bad_melon = dataset[dataset['好瓜'] == 0] plt.scatter(bad_melon['密度'],bad_melon['含糖率'],marker='o',color='r',s=100,label='bad') plt.scatter(good_melon['密度'],good_melon['含糖率'],marker='o',color='g',s=100,label='good')
![](https://i-blog.csdnimg.cn/blog_migrate/8f900a89c6347c561fdf2122f13be562.gif)
![](https://i-blog.csdnimg.cn/blog_migrate/961ddebeb323a10fe0623af514929fc1.gif)
import numpy as np import matplotlib.pyplot as plt data = [[0.697, 0.460, 1], [0.774, 0.376, 1], [0.634, 0.264, 1], [0.608, 0.318, 1], [0.556, 0.215, 1], [0.403, 0.237, 1], [0.481, 0.149, 1], [0.437, 0.211, 1], [0.666, 0.091, 0], [0.243, 0.267, 0], [0.245, 0.057, 0], [0.343, 0.099, 0], [0.639, 0.161, 0], [0.657, 0.198, 0], [0.360, 0.370, 0], [0.593, 0.042, 0], [0.719, 0.103, 0]] #数据集按瓜好坏分类 data = np.array([i[:-1] for i in data]) X0 = np.array(data[:8]) X1 = np.array(data[8:]) #求正反例均值 miu0 = np.mean(X0, axis=0).reshape((-1, 1)) miu1 = np.mean(X1, axis=0).reshape((-1, 1)) #求协方差 cov0 = np.cov(X0, rowvar=False) cov1 = np.cov(X1, rowvar=False) #求出w S_w = np.mat(cov0 + cov1) Omiga = S_w.I * (miu0 - miu1) #画出点、直线 plt.scatter(X0[:, 0], X0[:, 1], c='b', label='+', marker = '+') plt.scatter(X1[:, 0], X1[:, 1], c='r', label='-', marker = '_') plt.plot([0, 1], [0, -Omiga[0] / Omiga[1]], label='y') plt.xlabel('密度', fontproperties='SimHei', fontsize=15, color='green'); plt.ylabel('含糖率', fontproperties='SimHei', fontsize=15, color='green'); plt.title(r'LinearDiscriminantAnalysis', fontproperties='SimHei', fontsize=25); plt.legend() plt.show()
3.6
对于非线性可分的数据,要想使用判别分析,一般思想是将其映射到更高维的空间上,使它在高维空间上线性可分进一步使用判别分析。
3.7
3.8
理论上的(纠错输出码)ECOC码能理想纠错的重要条件是每个码位出错的概率相当,因为如果某个码位的错误率很高,会导致这位始终保持相同的结果,不再有分类作用,这就相当于全0或者全 1的分类器。
3.9
书中提到,对于OvR,MvM来说,由于对每个类进行了相同的处理,其拆解出的二分类任务中类别不平衡的影响会相互抵消,因此通常不需要专门处理。以ECOC编码为例,每个生成的二分类器会将所有样本分成较为均衡的二类,使类别不平衡的影响减小。当然拆解后仍然可能出现明显的类别不平衡现象,比如一个超级大类和一群小类。
3.10
数据集(密码rw81)