【机器学习（10）】模型评价：数据集划方法（留出法和交叉验证法）

数据集划分方法

1） 划分基本准则：保持训练集和验证集之间的 互斥性
        准则解释：测试样本尽量不在训练样本中出现，以保证验证集上的表现能代表模型的泛化能力（比如期末测试题上出的内容不是课上讲的原题）

2） 留出法：
        直接将数据集划分成两个互斥的集合，其中一个做训练集，一个做验证集
        常用划分比例：7:3、7.5:2.5、8:2
        存在的问题：随机取样对模型的影响（比如这一次考试随机抽取的题目都是会做的，而第二次抽取的又恰巧是我不会做的），这种情况下测试的结果并不能代表我的真实水平

3）交叉验证法（cv）
        将数据集划分为k个大小相似的互斥子集，每一次以k-1个子集做训练，1个子集做验证，训练k次，最终返回的是k次训练结果的均值，因此交叉验证法又称为k折交叉法（k-fold）

留出法进行数据划分及模型评价得分

这里直接进行数据的分割

from sklearn.model_selection import train_test_split
training, testing = train_test_split(df,test_size=0.25, random_state=1)
x_train = training.copy().drop(columns=['average_price','id'])
y_train = training.copy()['average_price']
x_test = testing.copy().drop(columns=['average_price','id'])
y_test = testing.copy()['average_price']
print(f'the shape of the training set is {x_train.shape}')
print(f'the shape of the testing set is {x_test.shape}')

–> 输出的结果为：（按照0.75:0.25进行分割数据）

the shape of the training set is (673, 7)
the shape of the testing set is (225, 7)

查看留出法验证集上模型的表现

import warnings
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
warnings.filterwarnings('ignore')
pipe_lm.fit(x_train,y_train)
y_predict = pipe_lm.predict(x_test)
print(f'mean squared error is: {mean_squared_error(y_test,y_predict)}')
print(f'mean absolute error is: {mean_absolute_error(y_test,y_predict)}')
print(f'R Squared is: {r2_score(y_test,y_predict)}')

–> 输出的结果为：

mean squared error is: 37995892.98761668
mean absolute error is: 4396.432366811368
R Squared is: 0.5719194373282056

交叉验证法进行数据划分及模型评价得分

from sklearn.model_selection import KFold
k = 10
kf = KFold(n_splits=k, shuffle=True)

mse = []
mae = []
r_s2 = []

for train_index, test_index in kf.split(df):  # 拆分
    x_train, x_test = x.loc[train_index], x.loc[test_index]
    y_train, y_test = y.loc[train_index], y.loc[test_index]
    pipe_lm.fit(x_train,y_train)
    y_predict = pipe_lm.predict(x_test)  # 模型原型 选择
    k_mse = mean_squared_error(y_test,y_predict)
    mse.append(k_mse)
    print(f'mean squared error is {k_mse}')
    k_mae = mean_absolute_error(y_test,y_predict)
    mae.append(k_mae)
    print(f'mean absolute error is {k_mae}')
    k_r_s2 = r2_score(y_test,y_predict)
    r_s2.append(k_r_s2)
    print(f'R Squared is {k_r_s2}')

–> 输出的结果为：（分十次进行结果输出）

mean squared error is 33944053.2775839
mean absolute error is 4091.3521198501926
R Squared is 0.5092029932534381
mean squared error is 35114434.65383525
mean absolute error is 4058.507125221178
R Squared is 0.5689849523694475
mean squared error is 18197156.68175975
mean absolute error is 3369.9995687271
R Squared is 0.776182527685076
mean squared error is 34535243.46299915
mean absolute error is 4192.820971274831
R Squared is 0.5779088133315295
mean squared error is 40233080.92316229
mean absolute error is 4097.433595440258
R Squared is 0.5424187136274303
mean squared error is 30563555.203029394
mean absolute error is 4036.1654178309395
R Squared is 0.599523380094302
mean squared error is 45418168.17460259
mean absolute error is 4849.161828850654
R Squared is 0.5349900325354779
mean squared error is 39047300.59608697
mean absolute error is 4684.974341683951
R Squared is 0.6127954527533661
mean squared error is 40672733.401221015
mean absolute error is 4738.685116862728
R Squared is 0.5256179983368192
mean squared error is 37635022.702838585
mean absolute error is 4099.567148872294
R Squared is 0.4709723577684112

最后取十次结果的平均值即可

import numpy as np
print(f'mean squared error is {np.array(mse).mean()}')
print(f'mean absolute error is {np.array(mae).mean()}')
print(f'R Squared is {np.array(r_s2).mean()}')

–> 输出的结果为：

mean squared error is 35536074.90771189
mean absolute error is 4221.866723461413
R Squared is 0.5718597221755297

对比结果

可以看出两种方法最后的模型评价的分中，r2基本上是一致的，但是使用交叉验证法的mae/mse结果要比留出法要低，可以认为交叉验证法能有效的控制随机取样对模型的影响