数据清洗总结——python数据分析

导入相应的库和数据

import numpy as np
import pandas as pd
import seaborn as sns
# read the data
df = pd.read_csv('sberbank.csv')

# shape and data types of the data
print(df.shape)
print(df.dtypes)
#select numeric columns
df_numeric=df.select_types(include=[np.number])
numeric_cols=df.numeric.columns.values
print(numeric_cols)

1、缺失数据

data.isnull()#查看所有缺失值
data.isnull().any()#获取含有缺失值的列
data.isnull().all()#获取全部为NA的列

抛弃缺失值

df.drop(in_missing,axis=0,index=True)

填充观察值

众数、均值、中位数、统一的数，如‘-1’

#下面是中位数填充
med=df['life_sq'].median()
print(med)
df['life_sq'].fillna(med)

2、重复值处理

pandas.DataFrame.drop_duplicates(self, subset=None, keep=’first’, inplace=False)

pandas.Series.drop_duplicates(self, keep=’first’, inplace=False)

3、值替换

import numpy as np
data.replace([-999,-1000], np.nan)
data.replace({-999: np.nan, -1000:0})

4、哑变量操作

df=pd.get_dummies(df['key'])