零基础入门金融风控-贷款违约预测之TASK2_探索性数据分析EDA

1、读取文件

train = pd.read_csv('train.csv')
testA = pd.read_csv('testA.csv')

2、基本了解

查看数据集的变量个数、类型、统计特征等。

train.head()
testA.head()
testA.shape
train.shape
train.columns
testA.columns
train.info() 
train.describe()

3、利用pandas_profiling生成数据报告

生成标准化的数据报告，可快速查看各个变量的值分布，缺失值，变量间相关性等基本信息，为EDA提供参考。

import pandas_profiling
pfr=pandas_profiling.ProfileReport(train)
pfr.to_file("./example.html")

4、查看数据集中特征缺失值、唯一等值

训练集共有22列特征有缺失值，打印每个特征的缺失值占比，绘制缺失特征和缺失率排序图。

null_number=train.isnull().any().sum()
print(null_number)
missing=train.isnull().sum()/len(train)
print(missing)
miss=missing[missing>0]
miss.sort_values(inplace=True)
miss.plot.bar()

查看训练集中测试集中特征属性只有一个值的特征，即policyCode。

one_value_fea=[col for col in train.columns if train[col].nunique()<=1]
one_value_fea
one_value_fea_test=[col for col in testA.columns if testA[col].nunique()<=1]
one_value_fea_test

5、区分不同类型的的特征

5.1、区分数值型特征和类别型特征

数值型特征有：[‘id’, ‘loanAmnt’, ‘term’, ‘interestRate’, ‘installment’, ‘employmentTitle’, ‘homeOwnership’, ‘annualIncome’, ‘verificationStatus’, ‘isDefault’, ‘purpose’, ‘postCode’, ‘regionCode’, ‘dti’, ‘delinquency_2years’, ‘ficoRangeLow’, ‘ficoRangeHigh’, ‘openAcc’, ‘pubRec’, ‘pubRecBankruptcies’, ‘revolBal’, ‘revolUtil’, ‘totalAcc’, ‘initialListStatus’, ‘applicationType’, ‘title’, ‘policyCode’, ‘n0’, ‘n1’, ‘n2’, ‘n3’, ‘n4’, ‘n5’, ‘n6’, ‘n7’, ‘n8’, ‘n9’, ‘n10’, ‘n11’, ‘n12’, ‘n13’, ‘n14’]
类别型特征有：[‘grade’, ‘subGrade’, ‘employmentLength’, ‘issueDate’, ‘earliesCreditLine’]

numerical=list(train.select_dtypes(exclude=['object']).columns)
numerical
category=list(filter(lambda x: x not in numerical,list(train.columns)))
category

5.2区分数值型变量中的连续变量和分类变量

连续变量有：[‘id’, ‘loanAmnt’, ‘interestRate’, ‘installment’, ‘employmentTitle’, ‘annualIncome’, ‘purpose’, ‘postCode’, ‘regionCode’, ‘dti’, ‘delinquency_2years’, ‘ficoRangeLow’, ‘ficoRangeHigh’, ‘openAcc’, ‘pubRec’, ‘pubRecBankruptcies’, ‘revolBal’, ‘revolUtil’, ‘totalAcc’, ‘title’, ‘n0’, ‘n1’, ‘n2’, ‘n3’, ‘n4’, ‘n5’, ‘n6’, ‘n7’, ‘n8’, ‘n9’, ‘n10’, ‘n13’, ‘n14’]

分类变量有：Out[9]:[‘term’, ‘homeOwnership’, ‘verificationStatus’, ‘isDefault’, ‘initialListStatus’, ‘applicationType’, ‘policyCode’, ‘n11’, ‘n12’]

def get_numerical_serial(data,feas):
    numerical_serial=[]
    numerical_noserial=[]
    for fea in feas:
        temp=data[fea].nunique()
        if temp<=10:
            numerical_noserial.append(fea)
            continue
        numerical_serial.append(fea)
    return numerical_serial,numerical_noserial
numerical_serial,numerical_noserial=get_numerical_serial(train,numerical)
numerical_serial
numerical_noserial

5.3数值类别型变量分析

以term举例：
3 606902
5 193098
Name: term, dtype: int64

train['term'].value_counts()

5.4数值连续型变量分析

查看各个数值连续型变量的分布

import seaborn as sns
f=pd.melt(train,value_vars=numerical_serial)
g=sns.FacetGrid(f,col="variable",col_wrap=2,sharex=False,sharey=False)
g=g.map(sns.distplot,"value")