数据分析学习-数据清洗7步法

Step1 : read csv

data_raw = pd.read_csv('../input/titanicdataset-traincsv/train.csv')
data_raw

Step2 : preview data

data_raw.info()
data_raw.describe(include='all')

Step3: check null value for every column

data1 = data_raw.copy(deep=True)
data1.isnull().sum()

Step4: complete null value

data1['Age'].fillna(data1['Age'].median(), inplace = True)
data1['Embarked'].fillna(data1['Embarked'].mode()[0], inplace = True)
data1.isnull().sum()

Step5: feature engineering

Step 5.1: delete some features

drop_column = ['PassengerId','Cabin', 'Ticket']
data1.drop(drop_column, axis=1, inplace = True)

Step 5.2: create new feature

data1['FamilySize'] = data1 ['SibSp'] + data1['Parch'] + 1
data1

data1['Title'] = data1['Name'].str.split(", ", expand=True)[1].str.split(".", expand=True)[0]
data1

data_check1['Title'] = data_check1['Name'].str.split(", ", expand=True)[1].str.split(".", expand=True)[0]

# 分箱操作
data1['FareCut'] = pd.qcut(data1['Fare'], 4)
data1['AgeCut'] = pd.cut(data1['Age'].astype(int), 6)
data1

Step6: encode for categories columns
Step 6.1: Sklearn LabelEncode

from sklearn.preprocessing import LabelEncoder
label = LabelEncoder()
data1['Sex_Code'] = label.fit_transform(data1['Sex'])
data1['Embarked_Code'] = label.fit_transform(data1['Embarked'])
data1['Title_Code'] = label.fit_transform(data1['Title'])
data1['AgeBin_Code'] = label.fit_transform(data1['AgeCut'])
data1['FareBin_Code'] = label.fit_transform(data1['FareCut'])
data1

Step 6.2: Pandas get_dummies

# get_dummies 将长 DataFrame 变为宽 DataFrame
pd.get_dummies(data1['Sex'])

Step 7: check for data cleaning

data1[data1_x_alg].info()
print('-'*50)
data1_dummy.info()