动手学数据分析_第二章:第一节:数据清洗及特征处理

• 之前关于数据分析的学习是用到什么查阅什么,没有系统的学习过,最近通过这个机会,再复习下关于之前学过的数据分析的一些知识,并完善一些数据分析的方法.最近一系列的数据分析的内容来源于DataWhale开源.涉及到的数据集等可从github(https://github.com/datawhalechina/hands-on-data-analysis)上获取到.

【回顾&引言】前面一章的内容大家可以感觉到我们主要是对基础知识做一个梳理，让大家了解数据分析的一些操作，主要做了数据的各个角度的观察。那么在这里，我们主要是做数据分析的流程性学习，主要是包括了数据清洗以及数据的特征处理，数据重构以及数据可视化。这些内容是为数据分析最后的建模和模型评价做一个铺垫。

开始之前，导入numpy、pandas包和数据

#加载所需的库
import pandas as pd
import numpy as np

#加载数据train.csv
df = pd.read_csv("train.csv")
df.shape

(891, 12)

2 第二章：数据清洗及特征处理

我们拿到的数据通常是不干净的，所谓的不干净，就是数据中有缺失值，有一些异常点等，需要经过一定的处理才能继续做后面的分析或建模，所以拿到数据的第一步是进行数据清洗，本章我们将学习缺失值、重复值、字符串和数据转换等操作，将数据清洗成可以分析或建模的样子。

2.1 缺失值观察与处理

我们拿到的数据经常会有很多缺失值，比如我们可以看到Cabin列存在NaN，那其他列还有没有缺失值，这些缺失值要怎么处理呢

2.1.1 任务一：缺失值观察

(1) 请查看每个特征缺失值个数
(2) 请查看Age， Cabin， Embarked列的数据
以上方式都有多种方式，所以大家多多益善

#写入代码

# 方法1
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB

#写入代码

# 方法2
df.isnull() # 显示的还是DF,是个判断条件,数据变成了True或False
df.isnull().sum()

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

#写入代码

df[['Age','Cabin','Embarked']].head(10) # 空值在DF中显示的是NaN

	Age	Cabin	Embarked
0	22.0	NaN	S
1	38.0	C85	C
2	26.0	NaN	S
3	35.0	C123	S
4	35.0	NaN	S
5	NaN	NaN	Q
6	54.0	E46	S
7	2.0	NaN	S
8	27.0	NaN	S
9	14.0	NaN	C

2.1.2 任务二：对缺失值进行处理

(1)处理缺失值一般有几种思路

(2) 请尝试对Age列的数据的缺失值进行处理

(3) 请尝试使用不同的方法直接对整张表的缺失值进行处理

#处理缺失值的一般思路： (1)
#提醒：可使用的函数有--->dropna函数与fillna函数

# 一般可用 均值或者特定的值 对空值进行填充

#写入代码 :  (2)

# 和之前一样, 里面是判断,外面是填充,以0值填充空值
df[df['Age']==None]=0   # 但是并没有填充上!! 
df.head(10)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S
5	6	0	3	Moran, Mr. James	male	NaN	0	0	330877	8.4583	NaN	Q
6	7	0	1	McCarthy, Mr. Timothy J	male	54.0	0	0	17463	51.8625	E46	S
7	8	0	3	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.0750	NaN	S
8	9	1	3	Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)	female	27.0	0	2	347742	11.1333	NaN	S
9	10	1	2	Nasser, Mrs. Nicholas (Adele Achem)	female	14.0	1	0	237736	30.0708	NaN	C

#写入代码 : (2)

df[df['Age'].isnull()] = 0  # 这种方式填充上了!
df.head(3)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S

#写入代码: (2)

df[df['Age'] == np.nan] = 0 # 填充成功
df.head(10)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S
5	0	0	0	0	0	0.0	0	0	0	0.0000	0	0
6	7	0	1	McCarthy, Mr. Timothy J	male	54.0	0	0	17463	51.8625	E46	S
7	8	0	3	Palsson, Master. Gosta Leonard	male	2.0	3	1	349909	21.0750	NaN	S
8	9	1	3	Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)	female	27.0	0	2	347742	11.1333	NaN	S
9	10	1	2	Nasser, Mrs. Nicholas (Adele Achem)	female	14.0	1	0	237736	30.0708	NaN	C

【思考】检索空缺值用np.nan,None以及.isnull()哪个更好，这是为什么？如果其中某个方式无法找到缺失值，原因又是为什么？

【回答】数值列读取数据后，空缺值的数据类型为float64所以用None一般索引不到，比较的时候最好用np.nan

# 写入代码 : (3)

df.dropna() # dropna()函数会删除包含空值的所有行

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
5	0	0	0	0	0	0.0	0	0	0	0.0000	0	0
6	7	0	1	McCarthy, Mr. Timothy J	male	54.0	0	0	17463	51.8625	E46	S
10	11	1	3	Sandstrom, Miss. Marguerite Rut	female	4.0	1	1	PP 9549	16.7000	G6	S
...	...	...	...	...	...	...	...	...	...	...	...	...
878	0	0	0	0	0	0.0	0	0	0	0.0000	0	0
879	880	1	1	Potter, Mrs. Thomas Jr (Lily Alexenia Wilson)	female	56.0	0	1	11767	83.1583	C50	C
887	888	1	1	Graham, Miss. Margaret Edith	female	19.0	0	0	112053	30.0000	B42	S
888	0	0	0	0	0	0.0	0	0	0	0.0000	0	0
889	890	1	1	Behr, Mr. Karl Howell	male	26.0	0	0	111369	30.0000	C148	C

360 rows × 12 columns

df.fillna(0) # fillna(填充值) 会将填充值替换空值,不删除数据 

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	0	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	0	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	0	S
...	...	...	...	...	...	...	...	...	...	...	...	...
886	887	0	2	Montvila, Rev. Juozas	male	27.0	0	0	211536	13.0000	0	S
887	888	1	1	Graham, Miss. Margaret Edith	female	19.0	0	0	112053	30.0000	B42	S
888	0	0	0	0	0	0.0	0	0	0	0.0000	0	0
889	890	1	1	Behr, Mr. Karl Howell	male	26.0	0	0	111369	30.0000	C148	C
890	891	0	3	Dooley, Mr. Patrick	male	32.0	0	0	370376	7.7500	0	Q

891 rows × 12 columns

【思考1】dropna和fillna有哪些参数，分别如何使用呢?

思考回答

DataFrame.dropna( axis=0, how=‘any’, thresh=None, subset=None, inplace=False)

• axis : 删除行 还是 删除列, 默认是删除行
  • axis=0或axis='index’删除含有缺失值的行
  • axis=1或axis='columns’删除含有缺失值的列
• how参数当我们至少有一个NA时，确定是否从DataFrame中删除行或列
  • how='all’或者how=‘any’。
  • how='all’时表示删除全是缺失值的行（列）
  • how='any’时表示删除只要含有缺失值的行（列）
• thresh=n表示保留至少含有n个非na数值的行
• subset定义要在哪些列中查找缺失值 例 subset=[‘Age’,‘Sex’]
• inplace表示直接在原DataFrame修改

fillna(inplace=False,method=None,limit,axis)

• inplace 是否在原DataFrame上修改
• method参数的取值：{‘pad’,‘ffill’,‘backfill’,‘bfill’,‘None’}
  • pad/ffill：用前一个非缺失值去填充该缺失值
    • backfill/bfill：用下一个非缺失值填充该缺失值
      • None：指定一个值去替换缺失值（缺失默认）
• limit=3 : 填充3个,limit表示填充的个数
• axis ; 按行填充还是按列填充 axis=1 : 按行填充

2.2 重复值观察与处理

由于这样那样的原因，数据中会不会存在重复值呢，如果存在要怎样处理呢

2.2.1 任务一：请查看数据中的重复值

#写入代码

df[df.duplicated()] # df.duplicated()是判断哪行数据是重复出现过的

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
17	0	0	0	0	0	0.0	0	0	0	0.0	0	0
19	0	0	0	0	0	0.0	0	0	0	0.0	0	0
26	0	0	0	0	0	0.0	0	0	0	0.0	0	0
28	0	0	0	0	0	0.0	0	0	0	0.0	0	0
29	0	0	0	0	0	0.0	0	0	0	0.0	0	0
...	...	...	...	...	...	...	...	...	...	...	...	...
859	0	0	0	0	0	0.0	0	0	0	0.0	0	0
863	0	0	0	0	0	0.0	0	0	0	0.0	0	0
868	0	0	0	0	0	0.0	0	0	0	0.0	0	0
878	0	0	0	0	0	0.0	0	0	0	0.0	0	0
888	0	0	0	0	0	0.0	0	0	0	0.0	0	0

176 rows × 12 columns

2.2.2 任务二：对重复值进行处理

(1)重复值有哪些处理方式呢？

(2)处理我们数据的重复值

方法多多益善

#重复值有哪些处理方式：

df = df.drop_duplicates()
df
# 去重后还有715行,原891行  上面那个可以看出重复数据有176行 891-176=715

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S
...	...	...	...	...	...	...	...	...	...	...	...	...
885	886	0	3	Rice, Mrs. William (Margaret Norton)	female	39.0	0	5	382652	29.1250	NaN	Q
886	887	0	2	Montvila, Rev. Juozas	male	27.0	0	0	211536	13.0000	NaN	S
887	888	1	1	Graham, Miss. Margaret Edith	female	19.0	0	0	112053	30.0000	B42	S
889	890	1	1	Behr, Mr. Karl Howell	male	26.0	0	0	111369	30.0000	C148	C
890	891	0	3	Dooley, Mr. Patrick	male	32.0	0	0	370376	7.7500	NaN	Q

715 rows × 12 columns

2.2.3 任务三：将前面清洗的数据保存为csv格式

#写入代码

df.to_csv('test_clear.csv')

2.3 特征观察与处理

我们对特征进行一下观察，可以把特征大概分为两大类：
数值型特征：Survived ，Pclass， Age ，SibSp， Parch， Fare，其中Survived， Pclass为离散型数值特征，Age，SibSp， Parch， Fare为连续型数值特征
文本型特征：Name， Sex， Cabin，Embarked， Ticket，其中Sex， Cabin， Embarked， Ticket为类别型文本特征，数值型特征一般可以直接用于模型的训练，但有时候为了模型的稳定性及鲁棒性会对连续变量进行离散化。文本型特征往往需要转换成数值型特征才能用于建模分析。

2.3.1 任务一：对年龄进行分箱（离散化）处理

(1) 分箱操作是什么？

(2) 将连续变量Age平均分箱成5个年龄段，并分别用类别变量12345表示

(3) 将连续变量Age划分为[0,5) [5,15) [15,30) [30,50) [50,80)五个年龄段，并分别用类别变量12345表示

(4) 将连续变量Age按10% 30% 50% 70% 90%五个年龄段，并用分类变量12345表示

(5) 将上面的获得的数据分别进行保存，保存为csv格式

#分箱操作是什么：

# 分箱操作是将连续型数值变量转换为几个类别的操作

#写入代码 

# (2)将连续变量Age平均分箱成5个年龄段，并分别用类别变量12345表示
df['AgeBand'] = pd.cut(df['Age'],5,labels=[1,2,3,4,5])
df.to_csv('test_ave.csv')
df.head(3)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	3
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	2

#写入代码

# (3)将连续变量Age划分为[0,5) [5,15) [15,30) [30,50) [50,80)五个年龄段，并分别用类别变量12345表示
df['AgeBand']=pd.cut(df['Age'],[0,5,15,30,50,80],labels=[1,2,3,4,5])
df.to_csv('test_cut.csv')
df.head(3)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	3
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	4
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3

#写入代码

# (4)将连续变量Age按10% 30% 50 70% 90%五个年龄段，并用分类变量12345表示
df['AgeBand'] = pd.qcut(df['Age'],[0,0.1,0.3,0.5,0.7,0.9],labels = [1,2,3,4,5])
df.to_csv('test_pr.csv')
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4

【参考】https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html

【参考】https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.qcut.html

2.3.2 任务二：对文本变量进行转换

(1) 查看文本变量名及种类
(2) 将文本变量Sex， Cabin ，Embarked用数值变量12345表示
(3) 将文本变量Sex， Cabin， Embarked用one-hot编码表示

#写入代码

# (1) unique() 和 nunique()方法() 查看Carbin的类别数量
df['Cabin'].unique() 

array([nan, 'C85', 'C123', 0, 'E46', 'G6', 'C103', 'D56', 'A6',
       'C23 C25 C27', 'D33', 'B30', 'B28', 'C83', 'F33', 'F G73', 'E31',
       'A5', 'D10 D12', 'D26', 'C110', 'B58 B60', 'E101', 'D47', 'B86',
       'F2', 'C2', 'B19', 'A7', 'C49', 'F4', 'B4', 'B80', 'A31', 'D36',
       'D15', 'C93', 'C78', 'D35', 'C87', 'B77', 'E67', 'B94', 'C125',
       'C99', 'C118', 'D7', 'B49', 'D', 'C22 C26', 'C65', 'E36', 'C54',
       'B57 B59 B63 B66', 'C7', 'E34', 'C32', 'B18', 'C124', 'C91', 'E40',
       'T', 'E33', 'D37', 'B35', 'E50', 'C82', 'B96 B98', 'E10', 'C52',
       'E44', 'A34', 'C104', 'C111', 'C92', 'E38', 'E12', 'E63', 'B37',
       'C30', 'D20', 'B79', 'E25', 'D46', 'B73', 'B38', 'B39', 'B22',
       'C86', 'C70', 'A16', 'C101', 'C68', 'A10', 'E68', 'B41', 'A20',
       'D19', 'D50', 'D9', 'A23', 'B50', 'A26', 'D48', 'E58', 'B71',
       'B51 B53 B55', 'D49', 'B5', 'B20', 'F G63', 'C62 C64', 'E24',
       'C90', 'C126', 'C45', 'E8', 'B101', 'C46', 'D30', 'E121', 'D11',
       'E77', 'B3', 'D6', 'B82 B84', 'D17', 'A36', 'B69', 'E49', 'D28',
       'E17', 'A24', 'C50', 'B42', 'C148'], dtype=object)

df['Cabin'].nunique()

135

# (1) value_counts()方法能够显示出类别数量和每个类别有多少
df['Embarked'].value_counts()

S    554
C    130
Q     28
0      1
Name: Embarked, dtype: int64

#写入代码

# (2) 将类别文本转换为12345
# 方法1 : replace -- 将['a'] 替换为-> [1]
df['Sex_num'] = df['Sex'].replace(['male','female'],[1,2])
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1

#写入代码

# (2) 方法2 : map
df['Sex_num2'] = df['Sex'].map({'male':1,'female':2})
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num	Sex_num2
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1	1.0
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2	2.0
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2	2.0
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2	2.0
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1	1.0

# (2) 方法3 : 使用sklearn.preprocessing的LabelEncoder
from sklearn.preprocessing import LabelEncoder
for feat in ['Cabin', 'Ticket']:
    lbl = LabelEncoder()  
    label_dict = dict(zip(df[feat].unique(), range(df[feat].nunique())))
    df[feat + "_labelEncode"] = df[feat].map(label_dict)
    df[feat + "_labelEncode"] = lbl.fit_transform(df[feat].astype(str))

df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num	Sex_num2	Cabin_labelEncode	Ticket_labelEncode
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1	1.0	135	409
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2	2.0	74	472
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2	2.0	135	533
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2	2.0	50	41
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1	1.0	135	374

# 注释掉其中的两行,仍能得到想要的结果,
# dict会自动构造好,这里是自己对应的字典,挺好的
for feat in ['Cabin', 'Ticket']:
    lbl = LabelEncoder()  
#     label_dict = dict(zip(df[feat].unique(), range(df[feat].nunique())))
#     df[feat + "_labelEncode"] = df[feat].map(label_dict)
    df[feat + "_labelEncode"] = lbl.fit_transform(df[feat].astype(str))

df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num	Sex_num2	Cabin_labelEncode	Ticket_labelEncode
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1	1.0	135	409
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2	2.0	74	472
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2	2.0	135	533
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2	2.0	50	41
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1	1.0	135	374

# 写入代码

# (3) 将类别文本转换为one-hot编码
for feat in ["Embarked"]:   
    # prefix=feat是在构造特征列的时候,在列名前添加上feat_
    x = pd.get_dummies(df[feat],prefix=feat) 
    df = pd.concat([df, x], axis=1)
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	...	Cabin_labelEncode	Ticket_labelEncode	0	C	Q	S	Embarked_0	Embarked_C	Embarked_Q	Embarked_S
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	...	135	409	0	0	0	1	0	0	0	1
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	...	74	472	0	1	0	0	0	1	0	0
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	...	135	533	0	0	0	1	0	0	0	1
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	...	50	41	0	0	0	1	0	0	0	1
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	...	135	374	0	0	0	1	0	0	0	1

5 rows × 25 columns

2.3.3 任务三：从纯文本Name特征里提取出Titles的特征(所谓的Titles就是Mr,Miss,Mrs等)

#写入代码  正则匹配 : extract()方法
df['Title'] =df.Name.str.extract('([A-Za-z]+)\.', expand=False)
df.head(3)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	...	Ticket_labelEncode	0	C	Q	S	Embarked_0	Embarked_C	Embarked_Q	Embarked_S	Title
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	...	409	0	0	0	1	0	0	0	1	Mr
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	...	472	0	1	0	0	0	1	0	0	Mrs
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	...	533	0	0	0	1	0	0	0	1	Miss

3 rows × 26 columns

# str.extract() 若不匹配返回NaN

第二章第一节 : 总结

• 1.学习了缺失值的处理中空值的表示方式 None /np.nan / isnull()

• 2. Dropna()和fillna()函数

  DataFrame.dropna( axis=0, how=‘any’, thresh=None, subset=None, inplace=False)

  • axis : 删除行 还是 删除列, 默认是删除行
    • axis=0或axis='index’删除含有缺失值的行
    • axis=1或axis='columns’删除含有缺失值的列
  • how参数当我们至少有一个NA时，确定是否从DataFrame中删除行或列
    • how='all’或者how=‘any’。
    • how='all’时表示删除全是缺失值的行（列）
    • how='any’时表示删除只要含有缺失值的行（列）
  • thresh=n表示保留至少含有n个非na数值的行
  • subset定义要在哪些列中查找缺失值 例 subset=[‘Age’,‘Sex’]
  • inplace表示直接在原DataFrame修改

  fillna(inplace=False,method=None,limit,axis)

  • inplace 是否在原DataFrame上修改
  • method参数的取值：{‘pad’,‘ffill’,‘backfill’,‘bfill’,‘None’}
    • pad/ffill：用前一个非缺失值去填充该缺失值
    • backfill/bfill：用下一个非缺失值填充该缺失值
    • None：指定一个值去替换缺失值（缺失默认）
  • limit=3 : 填充3个,limit表示填充的个数
  • axis ; 按行填充还是按列填充 axis=1 : 按行填充
• 3. 重复值的查看函数 df.duplicated() 和重复值的删除函数 df.drop_duplicates()
• 4. 对连续型数值变量的分箱操作
  • pd.cut(df[‘列名’],分箱依据,分箱后字段) / pd.pcut()按照百分比分箱
• 5. 类别型变量转换为数字表示
  • map / replace
  • 主要记住labelEncode : 这里面的字典可以自己定义 ,不指定会有默认的
• 6. str.extract() 正则提取