1、读取文件
train = pd.read_csv('train.csv')
testA = pd.read_csv('testA.csv')
2、基本了解
查看数据集的变量个数、类型、统计特征等。
train.head()
testA.head()
testA.shape
train.shape
train.columns
testA.columns
train.info()
train.describe()
3、利用pandas_profiling生成数据报告
生成标准化的数据报告,可快速查看各个变量的值分布,缺失值,变量间相关性等基本信息,为EDA提供参考。
import pandas_profiling
pfr=pandas_profiling.ProfileReport(train)
pfr.to_file("./example.html")
4、查看数据集中特征缺失值、唯一等值
训练集共有22列特征有缺失值,打印每个特征的缺失值占比,绘制缺失特征和缺失率排序图。
null_number=train.isnull().any().sum()
print(null_number)
missing=train.isnull().sum()/len(train)
print(missing)
miss=missing[missing>0]
miss.sort_values(inplace=True)
miss.plot.bar()
查看训练集中测试集中特征属性只有一个值的特征,即policyCode。
one_value_fea=[col for col in train.columns if train[col].nunique()<=1]
one_value_fea
one_value_fea_test=[col for col in testA.columns if testA[col].nunique()<=1]
one_value_fea_test
5、区分不同类型的的特征
5.1、区分数值型特征和类别型特征
数值型特征有:[‘id’, ‘loanAmnt’, ‘term’, ‘interestRate’, ‘installment’, ‘employmentTitle’, ‘homeOwnership’, ‘annualIncome’, ‘verificationStatus’, ‘isDefault’, ‘purpose’, ‘postCode’, ‘regionCode’, ‘dti’, ‘delinquency_2years’, ‘ficoRangeLow’, ‘ficoRangeHigh’, ‘openAcc’, ‘pubRec’, ‘pubRecBankruptcies’, ‘revolBal’, ‘revolUtil’, ‘totalAcc’, ‘initialListStatus’, ‘applicationType’, ‘title’, ‘policyCode’, ‘n0’, ‘n1’, ‘n2’, ‘n3’, ‘n4’, ‘n5’, ‘n6’, ‘n7’, ‘n8’, ‘n9’, ‘n10’, ‘n11’, ‘n12’, ‘n13’, ‘n14’]
类别型特征有:[‘grade’, ‘subGrade’, ‘employmentLength’, ‘issueDate’, ‘earliesCreditLine’]
numerical=list(train.select_dtypes(exclude=['object']).columns)
numerical
category=list(filter(lambda x: x not in numerical,list(train.columns)))
category
5.2区分数值型变量中的连续变量和分类变量
连续变量有:[‘id’, ‘loanAmnt’, ‘interestRate’, ‘installment’, ‘employmentTitle’, ‘annualIncome’, ‘purpose’, ‘postCode’, ‘regionCode’, ‘dti’, ‘delinquency_2years’, ‘ficoRangeLow’, ‘ficoRangeHigh’, ‘openAcc’, ‘pubRec’, ‘pubRecBankruptcies’, ‘revolBal’, ‘revolUtil’, ‘totalAcc’, ‘title’, ‘n0’, ‘n1’, ‘n2’, ‘n3’, ‘n4’, ‘n5’, ‘n6’, ‘n7’, ‘n8’, ‘n9’, ‘n10’, ‘n13’, ‘n14’]
分类变量有:Out[9]:[‘term’, ‘homeOwnership’, ‘verificationStatus’, ‘isDefault’, ‘initialListStatus’, ‘applicationType’, ‘policyCode’, ‘n11’, ‘n12’]
def get_numerical_serial(data,feas):
numerical_serial=[]
numerical_noserial=[]
for fea in feas:
temp=data[fea].nunique()
if temp<=10:
numerical_noserial.append(fea)
continue
numerical_serial.append(fea)
return numerical_serial,numerical_noserial
numerical_serial,numerical_noserial=get_numerical_serial(train,numerical)
numerical_serial
numerical_noserial
5.3数值类别型变量分析
以term举例:
3 606902
5 193098
Name: term, dtype: int64
train['term'].value_counts()
5.4数值连续型变量分析
查看各个数值连续型变量的分布
import seaborn as sns
f=pd.melt(train,value_vars=numerical_serial)
g=sns.FacetGrid(f,col="variable",col_wrap=2,sharex=False,sharey=False)
g=g.map(sns.distplot,"value")