整体流程
1.数据读取
2.特征理解分析
3.数据清洗与预处理
4.建立模型
准备工作
第一步:理解数据,分析每个字段的意思
PassengerID 乘客唯一编号(无关)
Pclass 船舱等级
Survived 是否获救
Sex 二值属性(female、male)
SibSp 一起上船的兄弟姐妹的个数
Parch 一起上船的老人和孩子的数量
Ticket 船票的编号(可能无关)
Fare 船票的价格
Cabin 住的船舱的编号(可能无关,不知道船舱和具体位置分布,是否靠窗?)
Embarked 在哪个码头上岸?SCQ码头(可能无关,做后这个名仕有关的,因为有的码头登上的有钱人比较多,他们购买一二等舱船票,或者登船的性别不均衡,而船舱等级和性别影响最后的获救几率,所以这个特征不能丢失)
第二步:导入相关的库:
1.numpy科学计算库
2.pandas数据分析处理库
3.matplotlib可视化库
4.seaborn可视化库,封装在matplotlib基础之上
5.scikit-learn机器学习库
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
#过滤掉警告
import warnings
warnings.filterwarnings("ignore")
#指定画图风格
plt.style.use("fivethirtyeight")
%matplotlib inline
数据分析操作
1.读取数据
data = pd.read_csv("train-Copy1.csv")
data.head()
默认显示前5行的数据,可以在head()的括号里面指定参数
2.统计每列缺失值的个数
data.isnull().sum()
可以看到Age的缺失值比较少(相对于这个数据量来说),可以进行缺失值填充;而Cabin这一列的缺失值太多了,有根据上面的分析跟是否获救好像没什么关系,所以直接将这一列扔掉
3.整体看看数据的一些统计值
data.describe()
这里过滤掉了一些有字符串数值的属性,因为字符串是不能计算平均值这些统计指标的。
4.绘图查看获救的比例
f, ax = plt.subplots(1, 2, figsize = (18, 8))
data['Survived'].value_counts().plot.pie(explode=[0, 0.1], autopct = '%1.1f%%', ax = ax[0],shadow = True)
ax[0].set_title('Survived')
ax[0].set_ylabel('')
sns.countplot('Survived', data = data, ax = ax[1])
ax[1].set_title('Survived')
plt.show()
可以看到最终只有38.4%的人获救
5.数据特征分析
离散值:性别(男、女)、登船地点(S Q C)
连续值:年龄、船票价格
查看性别与是否获救之间的关系
data.groupby(['Sex', 'Survived'])['Survived'].count()
可以看出船上的人男性明显比女性多,而获救的人当中,女性较多男性较少,说明性别对结果有较大的影响,也可以通过绘图更直观的查看
sns.countplot('Sex', hue = 'Survived', data = data)
plt.show()
查看船舱等级跟获救情况的关系
pd.crosstab(data.Pclass, data.Survived, margins = True).style.background_gradient(cmap = 'summer_r')
可以看出,船舱等级越高,获救的可能性越大,说明获救几率与船舱等级相关。
多特征分析:综合分析性别和船舱等级对结果的影响
pd.crosstab([data.Sex,data.Survived], data.Pclass, margins = True).style.background_gradient(cmap = 'summer_r')
结论:虽然女性的获救比例比较高,但是也分船舱等级,但是三等舱获救的女性比一等舱获救的男性数量都多,画图更直观:
sns.factorplot('Pclass', 'Survived', hue = 'Sex', data = data)
plt.show()
Age:连续值特征对结果的影响
f, ax = plt.subplots(1, 2, figsize = (18, 8))
sns.violinplot('Pclass', 'Age', hue = 'Survived', data = data, ax = ax[0])
ax[0].set_title('Pclass and Age vs Survived')
ax[0].set_yticks(range(0,110,10))
sns.violinplot('Sex', 'Age', hue = 'Survived', data = data, ax = ax[1])
ax[0].set_title('Sex and Age vs Survived')
ax[0].set_yticks(range(0,110,10))
plt.show()
结论:
(1)20-50岁获救几率高一些
(2)对男性来说,随着年龄的增长,存活率降低
6.异常值检测
from collections import Counter
def detect_outliers(df, n, features):
outlier_indices = []
for col in features:
Q1 = np.percentile(df[col], 25)#计算下四分位数
Q2 = np.percentile(df[col], 75)#计算上四分位数
IQR = Q2 - Q1
outlier_step = 1.5 * IQR
outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q2+ outlier_step)].index
outlier_indices.extend(outlier_list_col)
outlier_indices = Counter(outlier_indices)
mutiple_outliers = list(k for k,v in outlier_indices.items() if v > n)
return mutiple_outliers
outliers_to_drop = detect_outliers(data, 2,['Age', 'SibSp', 'Parch', 'Fare'])
data.loc[outliers_to_drop]#显示检测出来的异常值
data= data.drop(outliers_to_drop, axis = 0).reset_index(drop=True)
将上面检测出来的异常值丢掉
7.缺失值填充
- 平均值
- 经验值(一般不太靠谱,并不是每个值都适用)
- 回归模性预测
- 剔除掉
Age和Fare用中位数填充,Embarked用众数填充
#读入test数据集的数据
test_data = pd.read_csv("test-Copy1.csv")
#对训练数据集和测试数据集都进行缺失值填充
data_to_clean = [data, test_data]
for dataset in data_to_clean:
dataset['Age'].fillna(dataset['Age'].median(), inplace = True)
dataset['Embarked'].fillna('S', inplace = True)
dataset['Fare'].fillna(dataset['Fare'].median(), inplace = True)
8.特征的构建(特征工程)
(1)丢掉一些没用的属性列
drop_columns = ['PassengerId','Cabin', 'Ticket']
data.drop(drop_columns, axis = 1, inplace = True)
(2)构建一些特征,文本值进行一些处理
补:pd.cut()与pd.qcut()的区别:
pd.cut()主要用于对数据从最大值到最小值进行等距划分
pd.qcut()按照数据出现频率百分比划分
for dataset in data_to_clean:
#每个人是不是一个人来的
dataset['family_members'] = dataset['SibSp'] + dataset['Parch']
dataset['is_alone'] = 1
dataset['is_alone'].loc[dataset['family_members'] >= 1] = 0
#取出姓名的前缀
dataset['title'] = dataset['Name'].str.split(", ", expand=True)[1].str.split(".", expand=True)[0]
#将年龄和船票价格划分为几个区间
dataset['FareBin'] = pd.qcut(dataset['Fare'], 4)
dataset['AgeBin'] = pd.cut(dataset['Age'].astype(int), 5)
#名字前缀个数小于10的用‘Other’代替
title_min = 10
title_names = (data['title'].value_counts() < title_min)
data['title'] = data['title'].apply(lambda x: 'Other' if title_names.loc[x] == True else x)
print(data['title'].value_counts())
from sklearn.preprocessing import LabelEncoder
label = LabelEncoder()
#使用LabelEncoder对一些不连续的值进行编码
for dataset in data_to_clean:
dataset['sex_code'] = label.fit_transform(dataset['Sex'])
dataset['embarked_code'] = label.fit_transform(dataset['Embarked'])
dataset['title_code'] = label.fit_transform(dataset['title'])
dataset['agebin_code'] = label.fit_transform(dataset['AgeBin'])
dataset['farebin_code'] = label.fit_transform(dataset['FareBin'])
#原始的数据
data_x = ['Sex','Pclass', 'Embarked', 'title','SibSp', 'Parch', 'Age', 'Fare', 'family_members', 'is_alone']
data_x_use = ['sex_code','Pclass', 'embarked_code', 'title_code','SibSp', 'Parch', 'Age', 'Fare']
target = ['Survived']
data_xy = target + data_x
data_x_bin = ['sex_code','Pclass', 'embarked_code', 'title_code', 'family_members', 'agebin_code', 'farebin_code']
data_xy_bin = target + data_x_bin
#进行One-Hot编码
data_dummy = pd.get_dummies(data[data_x])
data_x_dummy = data_dummy.columns.tolist()
data_xy_dummy = target + data_x_dummy
data_dummy.head()
对Sex,Embarked,title进行One-Hot编码后的结果:
9.数据集的分离
#训练集测试集分离train_test_split
from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(data[data_x_use], data[target],random_state = 0)
train_x_bin, test_x_bin, train_y_bin, test_y_bin = train_test_split(data[data_x_bin], data[target] , random_state = 0)
train_x_dummy, test_x_dummy, train_y_dummy, test_y_dummy = train_test_split(data_dummy[data_x_dummy], data[target], random_state = 0)
print("Data Shape: {}".format(data.shape))
print("Train Shape: {}".format(train_x.shape))
print("Test Shape: {}".format(test_x.shape))
10.建模预测
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegressionCV
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier, ExtraTreeClassifier
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.model_selection import ShuffleSplit, cross_validate
import xgboost as xgb
algorithm = [
KNeighborsClassifier(),
LogisticRegressionCV(),
SVC(probability=True),
DecisionTreeClassifier(),
ExtraTreeClassifier(),
RandomForestClassifier(),
ExtraTreesClassifier(),
xgb.XGBClassifier()
]
cv_split = ShuffleSplit(n_splits = 10, test_size = .3, train_size = .6, random_state = 0)
algorithm_columns = ['algorithm name', 'algorithm parameters', 'algorithm train accuracy mean', 'algorithm test accuracy mean', 'algorithm time']
algorithm_compare = pd.DataFrame(columns = algorithm_columns)
algorithm_predict = data[target]
row_index = 0
for alg in algorithm:
algorithm_name = alg.__class__.__name__
algorithm_compare.loc[row_index, 'algorithm name'] = algorithm_name
algorithm_compare.loc[row_index, 'algorithm parameters'] = str(alg.get_params())
cv_results = cross_validate(alg, data[data_x_bin], data[target], cv = cv_split)
algorithm_compare.loc[row_index, 'algorithm time'] = cv_results['fit_time'].mean()
algorithm_compare.loc[row_index, 'algorithm train accuracy mean'] = cv_results['train_score'].mean()
algorithm_compare.loc[row_index, 'algorithm test accuracy mean'] = cv_results['test_score'].mean()
alg.fit(data[data_x_bin], data[target])
algorithm_predict[algorithm_name] = alg.predict(data[data_x_bin])
row_index += 1
algorithm_compare.sort_values(by = ['algorithm test accuracy mean'], ascending = False, inplace = True)
algorithm_compare
sns.barplot(x='algorithm test accuracy mean', y = 'algorithm name', data = algorithm_compare, color = 'm')
plt.title('Machine Learning Algorithm Accuracy Score \n')
plt.xlabel('Accuracy Score (%)')
plt.ylabel('Algorithm')