集成算法-随机森林与案例实战-泰坦尼克获救预测

集成算法-随机森林

Ensemble learning

目的：让机器学习效果更好，单个不行，群殴走起

Bagging：训练多个分类器取平均 $f(x)=1/M{\sum }_{m=1}^M{f}_m(x)$

Boosting：从弱学习器开始加强，通过加权来进行训练 $F_m(x)=F_{m-1}(x)+{\mathrm{argmin}}_h{\sum }_{i=1}^{n}L\left(y_i,F_{m-1}\left(x_i\right)+h\left(x_i\right)\right)$ （加入一棵树，要比原来强）

Stacking：聚合多个分类或回归模型（可以分阶段来做）

Bagging模型

全称： bootstrap aggregation（说白了就是并行训练一堆分类器）

最典型的代表就是随机森林啦；森林：很多个决策树并行放在一起；随机：数据采样随机，特征选择随机

随机森林

构造树模型：

由于二重随机性，使得每个树基本上都不会一样，最终的结果也会不一样。

树模型：

之所以要进行随机，是要保证泛化能力，如果树都一样，那就没意义了！

随机森林优势

它能够处理很高维度（feature很多）的数据，并且不用做特征选择；在训练完后，它能够给出哪些feature比较重要；容易做成并行化方法，速度比较快；可以进行可视化展示，便于分析。

KNN模型：

KNN就不太适合，因为很难去随机让泛化能力变强！所以基础模型用树模型就可以！

树模型：

理论上越多的树效果会越好，但实际上基本超过一定数量就差不多上下浮动了

Boosting模型

典型代表：AdaBoost， Xgboost
Adaboost会根据前一次的分类效果调整数据权重

解释：如果某一个数据在这次分错了，那么在下一次我就会给它更大的权重

最终的结果：每个分类器根据自身的准确性来确定各自的权重，再合体
Adaboost工作流程：每一次切一刀！最终合在一起。弱分类器这就升级了！

Stacking模型

堆叠：很暴力，拿来一堆直接上（各种分类器都来了）
可以堆叠各种各样的分类器（KNN,SVM,RF等等）

分阶段：第一阶段得出各自结果，第二阶段再用前一阶段结果训练

为了刷结果，不择手段！

堆叠在一起确实能使得准确率提升，但是速度是个问题。
集成算法是竞赛与论文神器，当我们更关注于结果时不妨来试试！

案例实战-泰坦尼克获救预测

import pandas #ipython notebook
titanic = pandas.read_csv("titanic_train.csv")
titanic.head(5)
# print (titanic.describe())

# age有些数据缺失，需要填充缺失值；用均值填充
titanic["Age"] = titanic["Age"].fillna(titanic["Age"].median())
print(titanic.describe())

   PassengerId    Survived      Pclass         Age       SibSp  \
count   891.000000  891.000000  891.000000  891.000000  891.000000   
mean    446.000000    0.383838    2.308642   29.361582    0.523008   
std     257.353842    0.486592    0.836071   13.019697    1.102743   
min       1.000000    0.000000    1.000000    0.420000    0.000000   
25%     223.500000    0.000000    2.000000   22.000000    0.000000   
50%     446.000000    0.000000    3.000000   28.000000    0.000000   
75%     668.500000    1.000000    3.000000   35.000000    1.000000   
max     891.000000    1.000000    3.000000   80.000000    8.000000   

            Parch        Fare  
count  891.000000  891.000000  
mean     0.381594   32.204208  
std      0.806057   49.693429  
min      0.000000    0.000000  
25%      0.000000    7.910400  
50%      0.000000   14.454200  
75%      0.000000   31.000000  
max      6.000000  512.329200  

# 有些量不是数值，比如性别，就无法用数据显示；所以把字符的量映射成数值
print(titanic["Sex"].unique())

# Replace all the occurences of male with the number 0.
titanic.loc[titanic["Sex"] == "male", "Sex"] = 0
titanic.loc[titanic["Sex"] == "female", "Sex"] = 1

[‘male’ ‘female’]

print(titanic["Embarked"].unique())
# 还有一种nan缺失值，需要填充；不是数值就没法用均值填充；那么就谁最多就填充哪个
titanic["Embarked"] = titanic["Embarked"].fillna('S')
titanic.loc[titanic["Embarked"] == "S", "Embarked"] = 0
titanic.loc[titanic["Embarked"] == "C", "Embarked"] = 1
titanic.loc[titanic["Embarked"] == "Q", "Embarked"] = 2

[‘S’ ‘C’ ‘Q’ nan]

# Import the linear regression class
# 线性回归算法
from sklearn.linear_model import LinearRegression
# Sklearn also has a helper that makes it easy to do cross validation
from sklearn.model_selection import KFold

# The columns we'll use to predict the target
predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

# Initialize our algorithm class
alg = LinearRegression()
# Generate cross validation folds for the titanic dataset.  It return the row indices corresponding to train and test.
# We set random_state to ensure we get the same splits every time we run this.
# 为Titanic数据集生成交叉验证折叠。它返回与训练和测试相对应的行索引
# 样本平均分成3份，3折交叉验证
kf = KFold(n_splits=3, shuffle=True, random_state=1)

# 交叉验证 划分训练集 验证集
predictions = []
for train, test in kf.split(titanic[predictions]):
    # The predictors we're using the train the algorithm.  Note how we only take the rows in the train folds.
    train_predictors = (titanic[predictors].iloc[train,:])
    # The target we're using to train the algorithm.
    train_target = titanic["Survived"].iloc[train]
    # Training the algorithm using the predictors and target.
    alg.fit(train_predictors, train_target)
    # We can now make predictions on the test fold
    test_predictions = alg.predict(titanic[predictors].iloc[test,:])
    predictions.append(test_predictions)

import numpy as np
 
# The predictions are in three separate numpy arrays. Concatenate them into one.
# We concatenate them on asix 0, as they only have one axis.
predictions = np.concatenate(predictions,axis=0)
 
# Map predictions to outcomes (only possible outcomes are 1 and 0)
predictions[predictions > .5] = 1  # 映射成分类结果 计算准确率
predictions[predictions <= .5] = 0
 
# 注意这一行与源代码有出入
accuracy = sum(predictions==titanic['Survived'])/len(predictions)
 
# 验证集的准确率
print(accuracy)

0.5342312008978676

# 逻辑回归
from sklearn import model_selection
from sklearn.linear_model import LogisticRegression
# Initialize our algorithm
alg = LogisticRegression(random_state=1)
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
scores = model_selection.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=3)
# Take the mean of the scores (because we have one for each fold)
print(scores.mean())

0.7957351290684623

titanic_test = pandas.read_csv("test.csv")
# 同样进行数据转换
titanic_test["Age"] = titanic_test["Age"].fillna(titanic["Age"].median())
titanic_test["Fare"] = titanic_test["Fare"].fillna(titanic_test["Fare"].median())
titanic_test.loc[titanic_test["Sex"] == "male", "Sex"] = 0 
titanic_test.loc[titanic_test["Sex"] == "female", "Sex"] = 1
titanic_test["Embarked"] = titanic_test["Embarked"].fillna("S")

titanic_test.loc[titanic_test["Embarked"] == "S", "Embarked"] = 0
titanic_test.loc[titanic_test["Embarked"] == "C", "Embarked"] = 1
titanic_test.loc[titanic_test["Embarked"] == "Q", "Embarked"] = 2

# 使用随机森林模型
from sklearn import model_selection
from sklearn.ensemble import RandomForestClassifier

# 选择特征
predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked"]

# Initialize our algorithm with the default paramters
# n_estimators is the number of trees we want to make
# min_samples_split is the minimum number of rows we need to make a split
# min_samples_leaf is the minimum number of samples we can have at the place where a tree branch ends (the bottom points of the tree)
# n_estimators这个参数是指构造几棵树；min_samples_split最小的切分点；min_samples_leaf叶子节点最小的个数
alg = RandomForestClassifier(random_state=1, n_estimators=10, min_samples_split=2, min_samples_leaf=1)
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
# 交叉验证
kf = model_selection.KFold(n_splits=3, shuffle=True, random_state=1)
# 得到的分值
scores = model_selection.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=kf)

# Take the mean of the scores (because we have one for each fold)
# 取平均值
print(scores.mean())

0.7957351290684626

# 优化模型；调节参数
alg = RandomForestClassifier(random_state=1, n_estimators=100, min_samples_split=4, min_samples_leaf=2)
# Compute the accuracy score for all the cross validation folds.  (much simpler than what we did before!)
kf = model_selection.KFold(n_splits=3, shuffle=True, random_state=1)
scores = model_selection.cross_val_score(alg, titanic[predictors], titanic["Survived"], cv=kf)

# Take the mean of the scores (because we have one for each fold)
print(scores.mean())

0.8215488215488215

# 优化达到瓶颈；额外增加特征
# Generating a familysize column
titanic["FamilySize"] = titanic["SibSp"] + titanic["Parch"]

# The .apply method generates a new series
# 假设名字长度与逃生有关
titanic["NameLength"] = titanic["Name"].apply(lambda x: len(x))

# 名字里面有些称呼。。将称谓进行特征提取
import re

# A function to get the title from a name.
def get_title(name):
    # Use a regular expression to search for a title.  Titles always consist of capital and lowercase letters, and end with a period.
    title_search = re.search(' ([A-Za-z]+)\.', name)
    # If the title exists, extract and return it.
    if title_search:
        return title_search.group(1)
    return ""

# Get all the titles and print how often each one occurs.
titles = titanic["Name"].apply(get_title)
print(pandas.value_counts(titles))

# Map each title to an integer.  Some titles are very rare, and are compressed into the same codes as other titles.
title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Dr": 5, "Rev": 6, "Major": 7, "Col": 7, "Mlle": 8, "Mme": 8, "Don": 9, "Lady": 10, "Countess": 10, "Jonkheer": 10, "Sir": 9, "Capt": 7, "Ms": 2}
for k,v in title_mapping.items():
    titles[titles == k] = v

# Verify that we converted everything.
print(pandas.value_counts(titles))

# Add in the title column.
titanic["Title"] = titles

Mr          517
Miss        182
Mrs         125
Master       40
Dr            7
Rev           6
Mlle          2
Col           2
Major         2
Lady          1
Mme           1
Jonkheer      1
Countess      1
Capt          1
Don           1
Ms            1
Sir           1
Name: Name, dtype: int64
1     517
2     183
3     125
4      40
5       7
6       6
7       5
10      3
8       3
9       2
Name: Name, dtype: int64

# 选择最有价值的特征
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
import matplotlib.pyplot as plt
predictors = ["Pclass", "Sex", "Age", "SibSp", "Parch", "Fare", "Embarked", "FamilySize", "Title", "NameLength"]

# Perform feature selection
selector = SelectKBest(f_classif, k=5)
selector.fit(titanic[predictors], titanic["Survived"])

# Get the raw p-values for each feature, and transform from p-values into scores
scores = -np.log10(selector.pvalues_)

# Plot the scores.  See how "Pclass", "Sex", "Title", and "Fare" are the best?
plt.bar(range(len(predictors)), scores)
plt.xticks(range(len(predictors)), predictors, rotation='vertical')
plt.show()

# Pick only the four best features.
predictors = ["Pclass", "Sex", "Fare", "Title"]

alg = RandomForestClassifier(random_state=1, n_estimators=50, min_samples_split=8, min_samples_leaf=4)

# 不同的分类器集成在一起
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

# The algorithms we want to ensemble.
# We're using the more linear predictors for the logistic regression, and everything with the gradient boosting classifier.
algorithms = [
    [GradientBoostingClassifier(random_state=1, n_estimators=25, max_depth=3), ["Pclass", "Sex", "Age", "Fare", "Embarked", "FamilySize", "Title",]],
    [LogisticRegression(random_state=1), ["Pclass", "Sex", "Fare", "FamilySize", "Title", "Age", "Embarked"]]
]

# Initialize the cross validation folds
kf = KFold(n_splits=3, shuffle=True, random_state=1)

predictions = []
for train, test in kf.split(titanic[predictions]):
    train_target = titanic["Survived"].iloc[train]
    full_test_predictions = []
    # Make predictions for each algorithm on each fold
    for alg, predictors in algorithms:
        # Fit the algorithm on the training data.
        alg.fit(titanic[predictors].iloc[train,:], train_target)
        # Select and predict on the test fold.  
        # The .astype(float) is necessary to convert the dataframe to all floats and avoid an sklearn error.
        test_predictions = alg.predict_proba(titanic[predictors].iloc[test,:].astype(float))[:,1]
        full_test_predictions.append(test_predictions)
    # Use a simple ensembling scheme -- just average the predictions to get the final classification.
    test_predictions = (full_test_predictions[0] + full_test_predictions[1]) / 2
    # Any value over .5 is assumed to be a 1 prediction, and below .5 is a 0 prediction.
    test_predictions[test_predictions <= .5] = 0
    test_predictions[test_predictions > .5] = 1
    predictions.append(test_predictions)

# Put all the predictions together into one array.
predictions = np.concatenate(predictions, axis=0)

# Compute accuracy by comparing to the training data.
accuracy = sum(predictions == titanic["Survived"]) / len(predictions)
print(accuracy)

0.5454545454545454

titles = titanic_test["Name"].apply(get_title)
# We're adding the Dona title to the mapping, because it's in the test set, but not the training set
title_mapping = {"Mr": 1, "Miss": 2, "Mrs": 3, "Master": 4, "Dr": 5, "Rev": 6, "Major": 7, "Col": 7, "Mlle": 8, "Mme": 8, "Don": 9, "Lady": 10, "Countess": 10, "Jonkheer": 10, "Sir": 9, "Capt": 7, "Ms": 2, "Dona": 10}
for k,v in title_mapping.items():
    titles[titles == k] = v
titanic_test["Title"] = titles
# Check the counts of each unique title.
print(pandas.value_counts(titanic_test["Title"]))

# Now, we add the family size column.
titanic_test["FamilySize"] = titanic_test["SibSp"] + titanic_test["Parch"]

1     240
2      79
3      72
4      21
7       2
6       2
10      1
5       1
Name: Title, dtype: int64

predictors = ["Pclass", "Sex", "Age", "Fare", "Embarked", "FamilySize", "Title"]

algorithms = [
    [GradientBoostingClassifier(random_state=1, n_estimators=25, max_depth=3), predictors],
    [LogisticRegression(random_state=1), ["Pclass", "Sex", "Fare", "FamilySize", "Title", "Age", "Embarked"]]
]

full_predictions = []
for alg, predictors in algorithms:
    # Fit the algorithm using the full training data.
    alg.fit(titanic[predictors], titanic["Survived"])
    # Predict using the test dataset.  We have to convert all the columns to floats to avoid an error.
    predictions = alg.predict_proba(titanic_test[predictors].astype(float))[:,1]
    full_predictions.append(predictions)

# The gradient boosting classifier generates better predictions, so we weight it higher.
predictions = (full_predictions[0] * 3 + full_predictions[1]) / 4
predictions

array([0.11608616, 0.47313163, 0.12527407, 0.1305029 , 0.52004311,
0.14392415, 0.63984342, 0.18138299, 0.6778233 , 0.12095368,
0.12054308, 0.21154493, 0.91148439, 0.10814111, 0.89180609,
0.87818912, 0.1650196 , 0.13910405, 0.53990167, 0.551889 ,
0.22474624, 0.53695164, 0.90686439, 0.39533671, 0.88099059,
0.10288684, 0.9105501 , 0.13737153, 0.31376951, 0.12631623,
0.11615631, 0.18430576, 0.54919197, 0.49415142, 0.42910392,
0.14215743, 0.50825222, 0.52417057, 0.13230705, 0.28222509,
0.11076129, 0.47224516, 0.09914069, 0.83492474, 0.90028243,
0.14946376, 0.31905677, 0.13752021, 0.89021874, 0.53830768,
0.36218205, 0.17945767, 0.83411368, 0.87864985, 0.17768223,
0.13743949, 0.10594413, 0.12307288, 0.12054944, 0.912131 ,
0.13115382, 0.15454243, 0.13002401, 0.66586834, 0.66161485,
0.87338584, 0.67298304, 0.29010609, 0.35826283, 0.84875193,
0.6621158 , 0.12688447, 0.55224753, 0.37379733, 0.91064833,
0.40981952, 0.12962877, 0.83764819, 0.15754935, 0.6621158 ,
0.68151841, 0.199957 , 0.20583641, 0.12054308, 0.1876183 ,
0.13090738, 0.65624963, 0.53041797, 0.65419471, 0.80328565,
0.53643959, 0.12053355, 0.89344775, 0.12962877, 0.29113646,
0.12308551, 0.86391073, 0.14611431, 0.58582725, 0.12192637,
0.90507833, 0.1487208 , 0.13752021, 0.1222069 , 0.62296229,
0.13079695, 0.1462909 , 0.13752021, 0.12968111, 0.17805571,
0.14320048, 0.65420052, 0.89686624, 0.67202899, 0.87958304,
0.14013126, 0.11761531, 0.69866896, 0.369716 , 0.86312987,
0.87861506, 0.12587932, 0.90367706, 0.12049219, 0.13752021,
0.56989533, 0.12588394, 0.63632568, 0.13347645, 0.13308618,
0.12658342, 0.51402225, 0.23707771, 0.10764469, 0.09812515,
0.12399516, 0.13310033, 0.16414568, 0.51965018, 0.12217545,
0.20839696, 0.905255 , 0.19234654, 0.16288954, 0.43252003,
0.10460664, 0.34145834, 0.1349816 , 0.47224516, 0.34281095,
0.9150228 , 0.13168701, 0.10605776, 0.48641364, 0.11240039,
0.12396799, 0.91030126, 0.57927986, 0.43252003, 0.51097235,
0.65419138, 0.57892305, 0.82284027, 0.12047252, 0.28604791,
0.58375961, 0.3031538 , 0.14606114, 0.90355986, 0.52205389,
0.12051582, 0.13260088, 0.12396504, 0.13162456, 0.13180211,
0.87544577, 0.87770527, 0.29748826, 0.83379914, 0.85394535,
0.15754935, 0.33614481, 0.90334825, 0.13752021, 0.91674518,
0.13621589, 0.85707224, 0.12265744, 0.1414593 , 0.13585733,
0.13547124, 0.2615505 , 0.49879649, 0.12640652, 0.72288221,
0.10727619, 0.85708385, 0.59003429, 0.16902874, 0.53774638,
0.64770495, 0.66407876, 0.6048431 , 0.87648409, 0.16545898,
0.26353344, 0.62898006, 0.16722597, 0.89151239, 0.12309417,
0.127611 , 0.12047645, 0.24831085, 0.79802557, 0.41149964,
0.30039786, 0.65422041, 0.21582703, 0.89844275, 0.12962877,
0.81556725, 0.13597309, 0.84397565, 0.12687469, 0.87848249,
0.59630913, 0.12492255, 0.65419471, 0.11396013, 0.14478507,
0.25316202, 0.89445045, 0.11621755, 0.13753447, 0.34495662,
0.12790127, 0.19280511, 0.1403001 , 0.81278248, 0.89757781,
0.8764603 , 0.8246372 , 0.32985937, 0.12054243, 0.33097517,
0.28960309, 0.88001689, 0.16051699, 0.86312987, 0.58951613,
0.74977708, 0.15427546, 0.39857128, 0.13320041, 0.12632896,
0.12051582, 0.13752021, 0.12962877, 0.83210887, 0.12687265,
0.10829711, 0.1268804 , 0.85114827, 0.65202653, 0.16804208,
0.12054308, 0.22527114, 0.12051582, 0.50825222, 0.14031173,
0.34621912, 0.13752021, 0.91585286, 0.63242282, 0.13162416,
0.85927351, 0.16037479, 0.12507105, 0.14380903, 0.17086699,
0.51978252, 0.66306258, 0.65419471, 0.64321196, 0.71428972,
0.10527009, 0.12049219, 0.36739267, 0.13162456, 0.12962877,
0.33858104, 0.59221078, 0.13162456, 0.50262011, 0.120044 ,
0.12221704, 0.78214546, 0.12631623, 0.33515934, 0.11974185,
0.11749083, 0.17758584, 0.12164696, 0.133141 , 0.65419471,
0.8211081 , 0.33592591, 0.67771416, 0.20869482, 0.42050239,
0.13919512, 0.13794091, 0.12051779, 0.61683764, 0.89999581,
0.67464103, 0.23647358, 0.1765905 , 0.1213635 , 0.18705969,
0.1222069 , 0.13464434, 0.16414568, 0.46045705, 0.90515213,
0.12485392, 0.86892053, 0.34709001, 0.14585039, 0.17320122,
0.81934824, 0.33252581, 0.13162416, 0.64291933, 0.12136688,
0.25679033, 0.15446923, 0.09326367, 0.21115849, 0.35136293,
0.17856622, 0.11747726, 0.14689857, 0.91334888, 0.33472757,
0.61877993, 0.16414568, 0.62126457, 0.16774844, 0.85266978,
0.89655915, 0.16545898, 0.24611915, 0.16006513, 0.70327315,
0.15827211, 0.85634043, 0.12054177, 0.13752021, 0.56988285,
0.10391274, 0.87788259, 0.86962861, 0.1305029 , 0.92015643,
0.15582768, 0.1309077 , 0.53155834, 0.89622127, 0.17553662,
0.15585283, 0.90915277, 0.16579567, 0.13122589, 0.87592006,
0.90842303, 0.48844403, 0.17292445, 0.19909204, 0.13524414,
0.13752021, 0.13987028, 0.5390601 , 0.5941846 , 0.16075475,
0.83407033, 0.12398808, 0.11946654, 0.14628145, 0.18797311,
0.38986125, 0.87810898, 0.56353165, 0.12784258, 0.1030095 ,
0.91276766, 0.14226138, 0.88795149, 0.12588196, 0.12914704,
0.90748673, 0.12667147, 0.91041256, 0.36696583, 0.30783741,
0.19327157, 0.15249874, 0.26395095, 0.65418889, 0.64831126,
0.65419471, 0.90753918, 0.56786184, 0.12962877, 0.85998813,
0.10048191, 0.12962877, 0.41631829])

问题报错解决

ModuleNotFoundError: No module named 'sklearn.cross_validation’
TypeError: init() got multiple values for argument 'n_splits’
https://blog.csdn.net/qq_35781239/article/details/100866176
https://www.cnblogs.com/xiaoyh/p/11321780.html

案例代码都是老版本，运行就有错误，解决完BUG之后，代码运行后的结果与老师的不一样，最有怀疑的两个结果集，一个是那个线性回归的准确率，还有一个是最后那个集成分类器的结果，准确度差了很多，感觉不应该存在这样的情况。。。。。反正也没咋听懂，以后遇到再深入学习。