Datawhale-数据分析-泰坦尼克-第二单元第一节

【回顾&引言】前面一章的内容大家可以感觉到我们主要是对基础知识做一个梳理,让大家了解数据分析的一些操作,主要做了数据的各个角度的观察。那么在这里,我们主要是做数据分析的流程性学习,主要是包括了数据清洗以及数据的特征处理,数据重构以及数据可视化。这些内容是为数据分析最后的建模和模型评价做一个铺垫。

开始之前,导入numpy、pandas包和数据
#加载所需的库
import numpy as np
import pandas as pd
#加载数据train.csv
train_data = pd.read_csv('train.csv')
train_data.columns
Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')
train_data.head(10)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
5603Moran, Mr. JamesmaleNaN003308778.4583NaNQ
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S
7803Palsson, Master. Gosta Leonardmale2.03134990921.0750NaNS
8913Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)female27.00234774211.1333NaNS
91012Nasser, Mrs. Nicholas (Adele Achem)female14.01023773630.0708NaNC

2 第二章:数据清洗及特征处理

我们拿到的数据通常是不干净的,所谓的不干净,就是数据中有缺失值,有一些异常点等,需要经过一定的处理才能继续做后面的分析或建模,所以拿到数据的第一步是进行数据清洗,本章我们将学习缺失值、重复值、字符串和数据转换等操作,将数据清洗成可以分析或建模的亚子。

2.1 缺失值观察与处理

我们拿到的数据经常会有很多缺失值,比如我们可以看到Cabin列存在NaN,那其他列还有没有缺失值,这些缺失值要怎么处理呢

2.1.1 任务一:缺失值观察

(1) 请查看每个特征缺失值个数
(2) 请查看Age, Cabin, Embarked列的数据
以上方式都有多种方式,所以大家多多益善

#写入代码
train_data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB
#写入代码train_data.isnull().sum()
PassengerId      0Survived         0Pclass           0Name             0Sex              0Age            177SibSp            0Parch            0Ticket           0Fare             0Cabin          687Embarked         2dtype: int64
#写入代码train_data[['Age','Cabin','Embarked']].head(6)
AgeCabinEmbarked
022.0NaNS
138.0C85C
226.0NaNS
335.0C123S
435.0NaNS
5NaNNaNQ
2.1.2 任务二:对缺失值进行处理

(1)处理缺失值一般有几种思路

(2) 请尝试对Age列的数据的缺失值进行处理

(3) 请尝试使用不同的方法直接对整张表的缺失值进行处理

#处理缺失值的一般思路:# 对于数值型:取0,或删除数据,获取平均值/最大值/最小值#提醒:可使用的函数有--->dropna函数与fillna函数train_data[train_data['Age']== None] =0train_data.head(6)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
5603Moran, Mr. JamesmaleNaN003308778.4583NaNQ
#写入代码train_data[train_data['Age'].isnull()]=0train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
#写入代码train_data[train_data['Age']==np.nan] =0train_data.head(6)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
5000000.00000.000000
#写入代码train_data.dropna().head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
6701McCarthy, Mr. Timothy Jmale54.0001746351.8625E46S
101113Sandstrom, Miss. Marguerite Rutfemale4.011PP 954916.7000G6S
111211Bonnell, Miss. Elizabethfemale58.00011378326.5500C103S
train_data.fillna(0).head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.25000S
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.92500S
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.05000S

【思考1】dropna和fillna有哪些参数,分别如何使用呢?

#思考回答#【思考1】# DataFrame.dropna(axis=0, how='any', thresh=None, subset=None, inplace=False)  丢弃# axis:0表示舍弃行,1表示舍弃列;# how: 默认为any,参数有any/all,any:只要一行/列有一个值为NA,就舍弃;all:一行/列全为NA,才将那一行/列舍弃# thresh:保留包含thresh=N个non-NA的行/列# subset: list在哪些列中查看是否有缺失值# inplace: 是否在原数据上操作。如果为真,返回None否则返回新的copy,去掉了缺失值。建议在使用时将全部的缺省参数都写上,便于快速理解# DataFrame.fillna(value=None, method=None, axis=None, inplace=False, limit=None, downcast=None) 填充# value:用来填充空位的值,一般为0# method: {‘backfill’, ‘bfill’, ‘pad’, ‘ffill’, None}, default None在列上操作#          ffill / pad: 使用前一个值来填充缺失值#          backfill / bfill :使用后一个值来填充缺失值# limit:填充的缺失值个数限制。应该不怎么用data=pd.DataFrame([[1,5,9,np.nan],[np.nan,3,7,np.nan],[6,np.nan,2,np.nan] ,[np.nan,np.nan,np.nan,np.nan],[1,2,3,np.nan]])data
0123
01.05.09.0NaN
1NaN3.07.0NaN
26.0NaN2.0NaN
3NaNNaNNaNNaN
41.02.03.0NaN
data.dropna()
0123
data.dropna(how='all')
0123
01.05.09.0NaN
1NaN3.07.0NaN
26.0NaN2.0NaN
41.02.03.0NaN
data.dropna(axis=1,how='all')
012
01.05.09.0
1NaN3.07.0
26.0NaN2.0
3NaNNaNNaN
41.02.03.0
data.dropna(thresh=3)
0123
01.05.09.0NaN
41.02.03.0NaN
data.fillna(0)
0123
01.05.09.00.0
10.03.07.00.0
26.00.02.00.0
30.00.00.00.0
41.02.03.00.0
data.fillna(method='ffill')
0123
01.05.09.0NaN
11.03.07.0NaN
26.03.02.0NaN
36.03.02.0NaN
41.02.03.0NaN
data.fillna(method='bfill',limit=1)
0123
01.05.09.0NaN
16.03.07.0NaN
26.0NaN2.0NaN
31.02.03.0NaN
41.02.03.0NaN

【思考2】检索空缺值用np.nan要比用None好,这是为什么?

# 思考回答#【思考2】数值列读取数据后,空缺值的数据类型为float64所以用None一般索引不到,比较的时候最好用np.nan

【参考】https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html

【参考】https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.fillna.html

2.2 重复值观察与处理

由于这样那样的原因,数据中会不会存在重复值呢,如果存在要怎样处理呢

2.2.1 任务一:请查看数据中的重复值
#写入代码train_data[train_data.duplicated()].head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
2.2.2 任务二:对重复值进行处理

(1)重复值有哪些处理方式呢?

(2)处理我们数据的重复值

方法多多益善

#重复值有哪些处理方式:#1.全部删除、#2.保留一条train_data = train_data.drop_duplicates()train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
#写入代码
2.2.3 任务三:将前面清洗的数据保存为csv格式
#写入代码train_data.to_csv('train_clear.csv')

2.3 特征观察与处理

我们对特征进行一下观察,可以把特征大概分为两大类:
数值型特征:Survived ,Pclass, Age ,SibSp, Parch, Fare,其中Survived, Pclass为离散型数值特征,Age,SibSp, Parch, Fare为连续型数值特征
文本型特征:Name, Sex, Cabin,Embarked, Ticket,其中Sex, Cabin, Embarked, Ticket为类别型文本特征,数值型特征一般可以直接用于模型的训练,但有时候为了模型的稳定性及鲁棒性会对连续变量进行离散化。文本型特征往往需要转换成数值型特征才能用于建模分析。

2.3.1 任务一:对年龄进行分箱(离散化)处理

(1) 分箱操作是什么?

(2) 将连续变量Age平均分箱成5个年龄段,并分别用类别变量12345表示

(3) 将连续变量Age划分为[0,5) [5,15) [15,30) [30,50) [50,80)五个年龄段,并分别用类别变量12345表示

(4) 将连续变量Age按10% 30% 50% 70% 90%五个年龄段,并用分类变量12345表示

(5) 将上面的获得的数据分别进行保存,保存为csv格式

#分箱操作是什么:train_data['AgeLevel'] = pd.cut(train_data['Age'],5,labels=[1,2,3,4,5])train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevel
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS2
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C3
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS2
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S3
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS3
#写入代码train_data.to_csv('train_5.csv')
train_data['AgeLevel'] = pd.cut(train_data['Age'],[0,5,15,30,50,80],labels=[1,2,3,4,5])train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevel
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS3
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C4
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS3
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S4
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS4
#写入代码train_data.to_csv('train_Age_5cut')
#写入代码train_data['AgeLevel'] = pd.cut(train_data['Age'],[0,0.1,0.3,0.5,0.7,0.9],labels=[1,2,3,4,5])train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevel
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNSNaN
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85CNaN
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNSNaN
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123SNaN
4503Allen, Mr. William Henrymale35.0003734508.0500NaNSNaN
train_data.to_csv('train_Age_5per')

【参考】https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html

【参考】https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html

2.3.2 任务二:对文本变量进行转换

(1) 查看文本变量名及种类
(2) 将文本变量Sex, Cabin ,Embarked用数值变量12345表示
(3) 将文本变量Sex, Cabin, Embarked用one-hot编码表示

#写入代码#方法一:value_counts()train_data['Sex'].value_counts()
male      577female    314Name: Sex, dtype: int64
#写入代码train_data['Cabin'].value_counts()
B96 B98            4G6                 4C23 C25 C27        4C22 C26            3F2                 3F33                3D                  3E101               3B57 B59 B63 B66    2C92                2C68                2E33                2C78                2F4                 2C93                2B18                2B58 B60            2B77                2E24                2D26                2C52                2C123               2B20                2D33                2C125               2C65                2B35                2E44                2C124               2D35                2                  ..B3                 1E46                1F G63              1C86                1B19                1A6                 1C70                1B42                1E31                1E49                1E68                1B73                1B37                1C90                1E34                1B82 B84            1A7                 1C32                1D7                 1D28                1D37                1C62 C64            1E63                1B86                1B38                1E40                1B69                1C111               1A19                1B4                 1Name: Cabin, Length: 147, dtype: int64
#写入代码train_data['Embarked'].value_counts()
S    644C    168Q     77Name: Embarked, dtype: int64
#方法二:unique()train_data['Sex'].unique()
array(['male', 'female'], dtype=object)
train_data['Sex'].nunique()
2
# 将类别文本转换为1,2,3,4,5...# 方法一:replacetrain_data['Sex_num'] = train_data['Sex'].replace(['male','female'],[1,2])train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevelSex_num
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNSNaN1
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85CNaN2
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNSNaN2
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123SNaN2
4503Allen, Mr. William Henrymale35.0003734508.0500NaNSNaN1
train_data['Embarked_Num'] = train_data['Embarked'].replace(['S','C','Q'],[1,2,3])train_data.head(3)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevelSex_numEmbarked_Num
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNSNaN11.0
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85CNaN22.0
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNSNaN21.0
# 方法二:maptrain_data['Sex_num'] = train_data['Sex'].map({'male':1,'female':2})train_data.head(3)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevelSex_numEmbarked_Num
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNSNaN11.0
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85CNaN22.0
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNSNaN21.0
#方法三: 使用sklearn.preprocessing的LabelEncoderfrom sklearn.preprocessing import LabelEncoderfor i in ['Cabin','Ticket']:    lb = LabelEncoder()    label_dict = dict(zip(train_data[i].unique(),range(train_data[i].nunique())))    train_data[i + '_LabelEncoder'] = train_data[i].map(label_dict)    train_data[i + '_LabelEncoder'] = lb.fit_transform(train_data[i].astype(str))train_data.head(3)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarkedAgeLevelSex_numEmbarked_NumCabin_LabelEncoderTicket_LabelEncoder
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNSNaN11.0147523
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85CNaN22.081596
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNSNaN21.0147669
# 将文本类别转换为one-hot编码#pandas的get_dummies方法for i in ['Age','Embarked']:    x = pd.get_dummies(train_data[i],prefix=i)    train_data = pd.concat([train_data,x],axis=1)train_data.head(3)
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFare...Age_65.0Age_66.0Age_70.0Age_70.5Age_71.0Age_74.0Age_80.0Embarked_CEmbarked_QEmbarked_S
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500...0000000001
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833...0000000100
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250...0000000001

3 rows × 108 columns

2.3.3 任务三:从纯文本Name特征里提取出Titles的特征(所谓的Titles就是Mr,Miss,Mrs等)
#写入代码train_data['Titles'] = train_data.Name.str.extract('([A-Za-z]+)\.',expand=False)train_data.head()
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFare...Age_66.0Age_70.0Age_70.5Age_71.0Age_74.0Age_80.0Embarked_CEmbarked_QEmbarked_STitles
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500...000000001Mr
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833...000000100Mrs
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250...000000001Miss
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000...000000001Mrs
4503Allen, Mr. William Henrymale35.0003734508.0500...000000001Mr

5 rows × 109 columns

#保存最终你完成的已经清理好的数据
train_data.to_csv('train_fin.csv')
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值