数据挖掘小组学习：数据探索和分析

1.1 读取数据

本次的数据集格式为.csv，利用pandas中的read_csv()函数读入数据集，encoding为gb18030：

import pandas as pd
users_data = pd.read_csv(data_path,encoding='gb18030')
users_data.head(5)  # 查看读取数据的前5行

1.2 探索数据

users_data.info()
print(f'数据集的shape为：{users_data.shape}')

输出如下：

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4754 entries, 0 to 4753
Data columns (total 89 columns):
custid                                        4754 non-null int64
trade_no                                      4754 non-null float64
bank_card_no                                  4754 non-null object
low_volume_percent                            4752 non-null float64
middle_volume_percent                         4752 non-null float64
take_amount_in_later_12_month_highest         4754 non-null int64
trans_amount_increase_rate_lately             4751 non-null float64
trans_activity_month                          4752 non-null float64
trans_activity_day                            4752 non-null float64
transd_mcc                                    4752 non-null float64
trans_days_interval_filter                    4746 non-null float64
trans_days_interval                           4752 non-null float64
regional_mobility                             4752 non-null float64
student_feature                               1756 non-null float64
repayment_capability                          4754 non-null int64
is_high_user                                  4754 non-null int64
number_of_trans_from_2011                     4752 non-null float64
first_transaction_time                        4752 non-null float64
historical_trans_amount                       4754 non-null int64
historical_trans_day                          4752 non-null float64
rank_trad_1_month                             4752 non-null float64
trans_amount_3_month                          4754 non-null int64
avg_consume_less_12_valid_month               4752 non-null float64
abs                                           4754 non-null int64
top_trans_count_last_1_month                  4752 non-null float64
avg_price_last_12_month                       4754 non-null int64
avg_price_top_last_12_valid_month             4650 non-null float64
reg_preference_for_trad                       4752 non-null object
trans_top_time_last_1_month                   4746 non-null float64
trans_top_time_last_6_month                   4746 non-null float64
consume_top_time_last_1_month                 4746 non-null float64
consume_top_time_last_6_month                 4746 non-null float64
cross_consume_count_last_1_month              4328 non-null float64
trans_fail_top_count_enum_last_1_month        4738 non-null float64
trans_fail_top_count_enum_last_6_month        4738 non-null float64
trans_fail_top_count_enum_last_12_month       4738 non-null float64
consume_mini_time_last_1_month                4728 non-null float64
max_cumulative_consume_later_1_month          4754 non-null int64
max_consume_count_later_6_month               4746 non-null float64
railway_consume_count_last_12_month           4742 non-null float64
pawns_auctions_trusts_consume_last_1_month    4754 non-null int64
pawns_auctions_trusts_consume_last_6_month    4754 non-null int64
jewelry_consume_count_last_6_month            4742 non-null float64
status                                        4754 non-null int64
source                                        4754 non-null object
first_transaction_day                         4752 non-null float64
trans_day_last_12_month                       4752 non-null float64
id_name                                       4478 non-null object
apply_score                                   4450 non-null float64
apply_credibility                             4450 non-null float64
query_org_count                               4450 non-null float64
query_finance_count                           4450 non-null float64
query_cash_count                              4450 non-null float64
query_sum_count                               4450 non-null float64
latest_query_time                             4450 non-null datetime64[ns]
latest_one_month_apply                        4450 non-null float64
latest_three_month_apply                      4450 non-null float64
latest_six_month_apply                        4450 non-null float64
loans_score                                   4457 non-null float64
loans_credibility_behavior                    4457 non-null float64
loans_count                                   4457 non-null float64
loans_settle_count                            4457 non-null float64
loans_overdue_count                           4457 non-null float64
loans_org_count_behavior                      4457 non-null float64
consfin_org_count_behavior                    4457 non-null float64
loans_cash_count                              4457 non-null float64
latest_one_month_loan                         4457 non-null float64
latest_three_month_loan                       4457 non-null float64
latest_six_month_loan                         4457 non-null float64
history_suc_fee                               4457 non-null float64
history_fail_fee                              4457 non-null float64
latest_one_month_suc                          4457 non-null float64
latest_one_month_fail                         4457 non-null float64
loans_long_time                               4457 non-null float64
loans_latest_time                             4457 non-null datetime64[ns]
loans_credit_limit                            4457 non-null float64
loans_credibility_limit                       4457 non-null float64
loans_org_count_current                       4457 non-null float64
loans_product_count                           4457 non-null float64
loans_max_limit                               4457 non-null float64
loans_avg_limit                               4457 non-null float64
consfin_credit_limit                          4457 non-null float64
consfin_credibility                           4457 non-null float64
consfin_org_count_current                     4457 non-null float64
consfin_product_count                         4457 non-null float64
consfin_max_limit                             4457 non-null float64
consfin_avg_limit                             4457 non-null float64
latest_query_day                              4450 non-null float64
loans_latest_day                              4457 non-null float64
dtypes: datetime64[ns](2), float64(71), int64(12), object(4)
memory usage: 3.2+ MB
数据集的shape为：(4754, 89)

由上面的信息可知，数据集共有4754条实例和88个特征，且大部分特征的非空实例数量较为完整。其中’status’为标签，代表用户的逾期状态，0代表未逾期，1代表逾期。

1.3 数据清洗¶

1.3.1 去除重复记录

print(f'去重前数据集的shape为：{users_data.shape}')
users_data.drop_duplicates('custid',inplace=True)
print(f'去重后数据集的shape为：{users_data.shape}')

输出为：

去重前数据集的shape为：(4754, 89)
去重后数据集的shape为：(4754, 89)

由上面的结果可知数据集没有重复记录。

1.3.2 删除无关特征

通过探索数据部分，这里选择的无关特征有:

irrelevant_features = ['custid','trade_no','bank_card_no','first_transaction_time','id_name','source','loans_latest_time','latest_query_time']
users_data.drop(irrelevant_features,axis=1,inplace=True)
users_data.head(5)

输出：
特可见征变为80个。

1.3.3 数据类型分析

由info()函数的内容可知，80个特征中，特征’reg_preference_for_trad’数据类型是object，有10个特征数据类型是int型，69个特征数据类型是float.

1.3.4 数据类型转换

这里对特征’reg_preference_for_trad’的数据类型进行转换，先看一下该特征的所有取值类别：

df_plot = users_data[['reg_preference_for_trad','status']].groupby('reg_preference_for_trad').count()
df_plot

由上面的结果可见，特征’reg_preference_for_trad’的取值类别有5种，将其转化为int类型，[‘境外’，‘一线城市’,‘二线城市’,‘三线城市’,‘其他城市’]对应为[6,4,3,2,1],数值越大，代表用户消费场景级别更高。

由info()函数的内容可知，有两条实例需要做缺失值处理；这里使用众数填补缺失值

a = users_data['reg_preference_for_trad'].mode()
print(a[0])
users_data['reg_preference_for_trad'].fillna(a[0],inplace=True)
users_data.info()

然后将数据类型进行转换：

reg = ['境外','一线城市','二线城市','三线城市','其他城市']
to_int = [6,4,3,2,1]
reg_to_int = dict(zip(reg,to_int))
def trans_reg(var):
    global reg_to_int
    new_var = reg_to_int[var]
    return new_var
users_data['reg_preference_for_trad'] = users_data['reg_preference_for_trad'].apply(trans_reg)

1.3.5 缺失值处理

# 找出哪些特征存在缺失值
users_data.isnull().any()

输出：

# 将student_feature的缺失值填补为0
users_data['student_feature'].fillna(0,inplace=True)
users_data.info()

这里将所有特征的缺失值补全为平均值：（在没有分析特征的前提下做这样处理并不好，只是为了尝试一下数据处理的操作）

# 获取所有特征名
col_names = users_data.columns.values.tolist()
for col in col_names:
    val = float(f'{users_data[col].mean():.2f}')
    users_data[col].fillna(val,inplace=True)