DataWhale动手学数据分析第二阶段笔记

第一节——数据的清洗以及特征处理

数据清洗和特征工程都是机器学习前的重要步骤，通常情况下我们拿到的数据都是不干净，有噪声和干扰项存在的。我们首要的工作就是讲这些数据清洗干净，即对这些数据进行合理的预处理，是我们的数据更适用于分析和建模。

# 导入所需的库
import numpy as np
import pandas as pd

# 读取数据
df = pd.read_csv('train.csv')

# 通过之前学的info方法，我们可以查询到数据的信息，其中就包括对每列缺失值的计数
# 方法1
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB

# 方法2
df.isnull().sum()

PassengerId      0
Survived         0
Pclass           0
Name             0
Sex              0
Age            177
SibSp            0
Parch            0
Ticket           0
Fare             0
Cabin          687
Embarked         2
dtype: int64

# 方法3
melted_df = df.melt()  
value_counts = melted_df[melted_df.value.isnull()].variable.value_counts()  
print(value_counts)

variable
Cabin       687
Age         177
Embarked      2
Name: count, dtype: int64

在Pandas中，melt方法用于将DataFrame从宽格式转换为长格式。宽格式是指数据在多个列中展开，而长格式是指数据被“熔化”成两列：一个标识变量（通常是原来的列名）和一个值变量（原来是那些列中的值）。

通过上面的代码我们已经对数据有了初步的了解与观察，现在让我们开始对这些缺失值进行处理。
对待缺失值的处理方式主要有以下几种：
1.删除含有缺失值的个案,直接删除含有缺失值的行或列。这种方法适用于缺失值较少，且对整体数据影响不大的情况。如果缺失不随机，可以根据个案赋予权重来进行删除。
2.插补缺失值，对数据中的缺失项进行插补操作，有均值插补，中位数插补，众数插补，回归插补，多重插补等多种插补方法.
3.如果在某些情况下缺失值是有一定意义的，也可以选择保留。

# 例如将年龄列的缺失值都填充为0
df[df['Age']==None] = 0
df.head(3)
# 其他方法:
# df[df['Age'].isnull()] = 0
# df[df['Age'] == np.nan] = 0
# df.dropna()
# df.fillna(0)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S

在比较的时候由于空缺值的数据类型为float64所以用None索引不到，比较的时候最好使用np.nan

# 查看数据中是否存在重复值
df[df.duplicated()]

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked

# 将清洗之后的数据保存
df.to_csv('test_clear.csv')

特征观察与处理

数值型的特征一般可以直接用于模型的训练，但有时候位了模型的稳定性和鲁棒性会对模型进行离散化(分箱处理)。文本型特征往往需要转换成数值型特征才能用于建模分析

# 将数据中的年龄划分为5个时间段
df['AgeBand'] = pd.cut(df['Age'], 5, labels=[1,2,3,4,5])
df.head(5)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	3
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	2
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	3
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	3

# 存储数据
df.to_csv('test_ave.csv')

# 将连续变量age划分为0,5] (5,15] (15,30] (30,50] (50,80]五个年龄段，并使用数字1～5表示
df['AgeBand'] = pd.cut(df['Age'], [0,5,15,30,50,80], labels = [1,2,3,4,5])
df.head(3)

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	3
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	4
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3

# 将上面得到的数据存储起来
df.to_csv('test_cut')

# 将连续变量age按照10%,30%,50%,70%,90%五个年龄段，并用分类变量12345表示
df['AgeBand'] = pd.qcut(df['Age'], [0,0.1,0.3,0.5,0.7,0.9], labels = [1,2,3,4,5])
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4

df.to_csv('test_percent.csv')

正如前面所说，对于非数值型变量，我们也应该将其转换成可以被模型学习的类型，最常用的是读热编码

# 我们首先查看文本变量名及其种类
# 方法1:values_counts()
df['Sex'].value_counts()

Sex
male      577
female    314
Name: count, dtype: int64

df['Cabin'].value_counts()

Cabin
B96 B98        4
G6             4
C23 C25 C27    4
C22 C26        3
F33            3
              ..
E34            1
C7             1
C54            1
E36            1
C148           1
Name: count, Length: 147, dtype: int64

# 方法2:也是我常用的方法，使用unique方法
df['Embarked'].unique()

array(['S', 'C', 'Q', nan], dtype=object)

# 我们可以尝试将文本类别转换成数字类别，比如性别
df['Sex_num'] = df['Sex'].replace(['male','female'], [1,2])
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1

# 除了replace，我们还可以使用map函数
df['Sex_num'] = df['Sex'].map({'male':1, 'female':2})
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1

Sklearn库作为著名的机器学习库，其中也有很多用来数据处理的方法，我们可以借助Sklearn库中的LabelEncoder

from sklearn.preprocessing import LabelEncoder
for feat in ['Cabin', 'Ticket']:
    lbl = LabelEncoder()
    label_dict = dict(zip(df[feat].unique(), range(df[feat].nunique())))
    df[feat+'_labelEncode'] = df[feat].map(label_dict)
    df[feat+'_labelEncode'] = lbl.fit_transform(df[feat].astype(str))

df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked	AgeBand	Sex_num	Cabin_labelEncode	Ticket_labelEncode
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	NaN	S	2	1	147	523
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	C85	C	5	2	81	596
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	NaN	S	3	2	147	669
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	C123	S	4	2	55	49
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	NaN	S	4	1	147	472

# 将文本转化为one-hot编码
# 方法1:OneHotEncoder
for feat in ["Age", "Embarked"]:
    x = pd.get_dummies(df[feat], prefix=feat)
    df = pd.concat([df, x], axis=1)

df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	...	Age_65.0	Age_66.0	Age_70.0	Age_70.5	Age_71.0	Age_74.0	Age_80.0	Embarked_C	Embarked_Q	Embarked_S
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	...	False	False	False	False	False	False	False	False	False	True
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	...	False	False	False	False	False	False	False	True	False	False
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	...	False	False	False	False	False	False	False	False	False	True
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	...	False	False	False	False	False	False	False	False	False	True
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	...	False	False	False	False	False	False	False	False	False	True

5 rows × 107 columns

# 从文本Name特征中提取出Titles的特征
df['Title'] = df.Name.str.extract('([A-Za-z]+)\.', expand=False)
df.head()

	PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	...	Age_66.0	Age_70.0	Age_70.5	Age_71.0	Age_74.0	Age_80.0	Embarked_C	Embarked_Q	Embarked_S	Title
0	1	0	3	Braund, Mr. Owen Harris	male	22.0	1	0	A/5 21171	7.2500	...	False	False	False	False	False	False	False	False	True	Mr
1	2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Th...	female	38.0	1	0	PC 17599	71.2833	...	False	False	False	False	False	False	True	False	False	Mrs
2	3	1	3	Heikkinen, Miss. Laina	female	26.0	0	0	STON/O2. 3101282	7.9250	...	False	False	False	False	False	False	False	False	True	Miss
3	4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35.0	1	0	113803	53.1000	...	False	False	False	False	False	False	False	False	True	Mrs
4	5	0	3	Allen, Mr. William Henry	male	35.0	0	0	373450	8.0500	...	False	False	False	False	False	False	False	False	True	Mr

5 rows × 108 columns

df.to_csv('test_fin.csv')