在数据探索和特征工程阶段,仅仅使用了主表的数据:
主要包含一些客户的详细信息.
- application_train.csv
- application_test.csv
在数据探索阶段,对数据进行过一些重编码和对齐之后,新数据的特征列有之前的121,增加至241,包含TARGET.
Feature Engineering
Andrew Ng老师曾说过:“applied machine learning is basically feature engineering.” ,不管选择什么样的模型,好的特征工程总能使模型表现的更出色.
特征工程:
- 特征构造:在原有的数据基础上构造新的特征
- 选择特征:选择重要的特征或数据降维
下面会简单使用两种特征构造的方法:
- Ploynomial features
- Domain knowledge features
Polynomial features
多项式特征:这是一种非常简单的特征构造方法,基于数据原有的特征,构造出新的特征,比如:EXT_SOURCE_1^2
, EXT_SOURCE_2^2
, EXT_SOURCE_1
x EXT_SOURCE_2
, EXT_SOURCE_1
x EXT_SOURCE_2^2
, EXT_SOURCE_1^2
x EXT_SOURCE_2^2
,等等.由多个单独变量组合构造新的特征.
为什么这么做呢?
因为单个变量可能对TARGET的影响很小,但是多个变量的组合特征可能会增加对TARGET的影响,捕捉到变量之间的相互作用.关于多项式特征polynomial features in his excellent book Python for Data Science的一些方法总结.下面会用EXT_SOURCE和DAYS_BIRTH来构造一些多项式特征,Scikit-learn提供一个PolynomialFeatures的类,可以很方便的构造多项式特征,在创建PloynomialFeatures实例时需要传入一个参数degree,构造特征的数量会随着degree呈指数级的增长,为了避免模型出现过拟合,一定要把握好这个degree度.
import pandas as pd
from sklearn.preprocessing import Imputer
from sklearn.preprocessing import PolynomialFeatures
import warnings
warnings.filterwarnings('ignore')
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# load data, 上次数据探索后处理过的数据
train_data = pd.read_csv('data/recode_train_data.csv')
test_data = pd.read_csv('data/recode_test_data.csv')
train_data.shape, test_data.shape
((307511, 241), (48744, 240))
poly_features =train_data[['EXT_SOURCE_1','EXT_SOURCE_2','EXT_SOURCE_3','DAYS_BIRTH']]
poly_features_test =test_data[['EXT_SOURCE_1','EXT_SOURCE_2','EXT_SOURCE_3','DAYS_BIRTH']]
target = train_data['TARGET']
# 中位数填充缺失值
imputer = Imputer(strategy = 'median')
poly_features = imputer.fit_transform(poly_features)
poly_features_test = imputer.transform(poly_features_test)
# 实例
poly_transformer = PolynomialFeatures(degree = 3)
poly_transformer.fit(poly_features)
#
poly_features = poly_transformer.transform(poly_features)
poly_features_test = poly_transformer.transform(poly_features_test)
# 构造多项式特征后的特征数量
print(poly_features.shape[1])
35
新特征的名子,get_feature_names()
poly_features_names = poly_transformer.get_feature_names(input_features = ['EXT_SOURCE_1',
'EXT_SOURCE_2',
'EXT_SOURCE_3',
'DAYS_BIRTH'])
poly_features_names
['1',
'EXT_SOURCE_1',
'EXT_SOURCE_2',
'EXT_SOURCE_3',
'DAYS_BIRTH',
'EXT_SOURCE_1^2',
'EXT_SOURCE_1 EXT_SOURCE_2',
'EXT_SOURCE_1 EXT_SOURCE_3',
'EXT_SOURCE_1 DAYS_BIRTH',
'EXT_SOURCE_2^2',
'EXT_SOURCE_2 EXT_SOURCE_3',
'EXT_SOURCE_2 DAYS_BIRTH',
'EXT_SOURCE_3^2',
'EXT_SOURCE_3 DAYS_BIRTH',
'DAYS_BIRTH^2',
'EXT_SOURCE_1^3',
'EXT_SOURCE_1^2 EXT_SOURCE_2',
'EXT_SOURCE_1^2 EXT_SOURCE_3',
'EXT_SOURCE_1^2 DAYS_BIRTH',
'EXT_SOURCE_1 EXT_SOURCE_2^2',
'EXT_SOURCE_1 EXT_SOURCE_2 EXT_SOURCE_3',
'EXT_SOURCE_1 EXT_SOURCE_2 DAYS_BIRTH',
'EXT_SOURCE_1 EXT_SOURCE_3^2',
'EXT_SOURCE_1 EXT_SOURCE_3 DAYS_BIRTH',
'EXT_SOURCE_1 DAYS_BIRTH^2',
'EXT_SOURCE_2^3',
'EXT_SOURCE_2^2 EXT_SOURCE_3',
'EXT_SOURCE_2^2 DAYS_BIRTH',
'EXT_SOURCE_2 EXT_SOURCE_3^2',
'EXT_SOURCE_2 EXT_SOURCE_3 DAYS_BIRTH',
'EXT_SOURCE_2 DAYS_BIRTH^2',
'EXT_SOURCE_3^3',
'EXT_SOURCE_3^2 DAYS_BIRTH',
'EXT_SOURCE_3 DAYS_BIRTH^2',
'DAYS_BIRTH^3']
之前的相关性分析当中,'EXT_SOURCE’这些外部数据源跟TARGET是呈负相关的,下面计算一下这些弱相关特征的组合特征跟TARGET的相关性系数.
poly_features = pd.DataFrame(poly_features, columns = poly_features_names)
# 加上TARGET
poly_features['TARGET'] = target
# 计算相关系数与TARGET
poly_corrs = poly_features.corr()['TARGET'].sort_values()
相关性从小到大排序
poly_corrs = poly_corrs.drop(['TARGET'])
plt.figure(figsize = (10, 10))
poly_corrs.plot(kind='barh')
<matplotlib.axes._subplots.AxesSubplot at 0x7f9e7c757cf8>
poly_corrs.tail(10)
EXT_SOURCE_1 -0.098887
EXT_SOURCE_1^2 DAYS_BIRTH -0.097507
EXT_SOURCE_1 DAYS_BIRTH^2 -0.094913
EXT_SOURCE_1^2 -0.091034
EXT_SOURCE_1^3 -0.083005
DAYS_BIRTH -0.078239
DAYS_BIRTH^2 -0.076672
DAYS_BIRTH^3 -0.074273
TARGET 1.000000
1 NaN
Name: TARGET, dtype: float64
有些新构造的特征比原始的特征相关性更强一点,但是在开始建模时,新特征并不会全加到训练数据当中,我们可以适当的选择一些特征丢掉一些特征.实际在机器学习项目中,往往不容易选择,唯一的选择就是多做些尝试,有时候很难确定这个特征就比另一个好.
poly_features['SK_ID_CURR'] = train_data['SK_ID_CURR']
# 合并到train_data
poly_train_data = train_data.merge(poly_features, on = 'SK_ID_CURR', how = 'left')
# 合并poly_features_test 到 test_data
poly_features_test = pd.DataFrame(poly_features_test, columns = poly_features_names)
poly_features_test['SK_ID_CURR'] = test_data['SK_ID_CURR']
poly_test_data = test_data.merge(poly_features_test, on = 'SK_ID_CURR', how = 'left')
# 数据列对齐
poly_train_data, poly_test_data = poly_train_data.align(poly_test_data, join = 'inner',axis =1)
print('Train data shape:',poly_train_data.shape)
print('Test data shape:',poly_test_data.shape)
Train data shape: (307511, 274)
Test data shape: (48744, 274)
poly_train_data['TARGET'] = target
poly_train_data.shape
(307511, 275)
Save data
poly_train_data.to_csv('data/poly_train_data.csv',index = False)
poly_test_data.to_csv('data/poly_test_data.csv', index = False)
Domain Knowledge Features
领域知识特征或者专业知识特征,可能这个称呼不一定正确,先这么叫着,因为不是专业人士,金融知识相对有限,但是我们可以利用这些有限的知识构造一些特征.构造一些我们主观上认为可能会影响到客户是否会违约的特征.
下面是4个DKfeatures:
CREDIT_INCOME_PERCENT
: 信用额度占客户收入的百分比ANNUITY_INCOME_PERCENT
: 贷款年金占客户收入的百分比ANNUITY_CREDIT_PERCENT
: 年金占信用额度百分比DAYS_EMPLOYED_PERCENT
: 客户工作天数占年龄的百分比
domain_train_data = train_data.copy()
domain_test_data = test_data.copy()
domain_train_data['CREDIT_INCOME_PERCENT'] = domain_train_data['AMT_CREDIT']/domain_train_data['AMT_INCOME_TOTAL']
domain_train_data['ANNUITY_INCOME_PERCENT'] = domain_train_data['AMT_ANNUITY']/domain_train_data['AMT_INCOME_TOTAL']
domain_train_data['ANNUITY_CREDIT_PERCENT'] = domain_train_data['AMT_ANNUITY']/domain_train_data['AMT_CREDIT']
domain_train_data['DAYS_EMPLOYED_PERCENT'] = domain_train_data['DAYS_EMPLOYED']/domain_train_data['DAYS_BIRTH']
domain_test_data['CREDIT_INCOME_PERCENT'] = domain_test_data['AMT_CREDIT'] /domain_test_data['AMT_INCOME_TOTAL']
domain_test_data['ANNUITY_INCOME_PERCENT'] = domain_test_data['AMT_ANNUITY'] /domain_test_data['AMT_INCOME_TOTAL']
domain_test_data['ANNUITY_CREDIT_PERCENT'] = domain_test_data['AMT_ANNUITY'] /domain_test_data['AMT_CREDIT']
domain_test_data['DAYS_EMPLOYED_PERCENT'] = domain_test_data['DAYS_EMPLOYED'] /domain_test_data['DAYS_BIRTH']
相关性分析
domain_features = domain_train_data[['TARGET','AMT_CREDIT','AMT_ANNUITY','DAYS_EMPLOYED',
'DAYS_BIRTH','AMT_INCOME_TOTAL','CREDIT_INCOME_PERCENT',
'ANNUITY_INCOME_PERCENT','ANNUITY_CREDIT_PERCENT',
'DAYS_EMPLOYED_PERCENT']]
domain_corrs = domain_features.corr()['TARGET'].sort_values()
domain_corrs = domain_corrs.drop(['TARGET'])
plt.figure(figsize = (10, 6))
plt.xticks(rotation = 90)
plt.ylim(-0.1,0.2)
domain_corrs.plot.bar()
从相关性系数来看新构造的特征比原来的特征更强(除了DAYS_EMPLOYED),新特征还是对目标有一定的影响力的.
KDE核密度估计
从客户的不同人群中,观察这些特征
plt.figure(figsize = (12, 20))
for i, f_name in enumerate(['CREDIT_INCOME_PERCENT','ANNUITY_INCOME_PERCENT',
'ANNUITY_CREDIT_PERCENT','DAYS_EMPLOYED_PERCENT']):
plt.subplot(4, 1, i+1)
# TARGET = 0
sns.kdeplot(domain_features.loc[domain_features['TARGET'] == 0, f_name], label = 'target=0')
# TARGET = 1
sns.kdeplot(domain_features.loc[domain_features['TARGET'] == 1, f_name], label = 'target=1')
plt.title('Distribution of %s by TARGET'%f_name)
plt.xlabel('%s'%f_name)
plt.ylabel('Density')
# 调整子图之间的上下间距
plt.tight_layout(h_pad = 2.5)
前两个特征在不同人群中,密度估计曲线的拟合程度是很高的,对两类客户的区分度基本是没有,后面两个有明显的差异但是整体的趋势是类似的.仅从相关性和密度估计,也很难确定,这些特征对机器学习建模有多大的帮助,只有试试才知道.
补充一下Featuretool
Featuretools
feature LAB推出了一款特征生成工具,自动构造特征,
import featuretools as ft
auto_train_data = train_data.copy()
auto_test_data = test_data.copy()
target = train_data['TARGET']
auto_train_data.shape, auto_test_data.shape
((307511, 241), (48744, 240))
# initialize an EntitySet and give it an id
es = ft.EntitySet(id = 'train_data')
#load dataframe as an entity
es = es.entity_from_dataframe(entity_id = 'train_data', dataframe = auto_train_data,
index = 'SK_ID_CURR')
es
Entityset: train_data
Entities:
train_data [Rows: 307511, Columns: 241]
Relationships:
No relationships
auto_train_data, features = ft.dfs(entityset = es,target_entity='train_data',verbose=True)
Built 240 features
Elapsed: 00:08 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks
es2 = ft.EntitySet(id = 'test_data')
es2 = es2.entity_from_dataframe(entity_id = 'test_data', dataframe = auto_test_data,
index = 'SK_ID_CURR')
auto_test_data, features = ft.dfs(entityset = es2,target_entity='test_data',verbose=True)
Built 239 features
Elapsed: 00:03 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks
# 数据对齐
auto_train_data, auto_test_data = auto_train_data.align(auto_test_data, join = 'inner', axis =1)
auto_train_data.shape
(307511, 238)
auto_test_data.shape
(48744, 238)
auto_train_data['TARGET'] = target
auto_test_data.to_csv('data/auto_test_data.csv')
auto_train_data.to_csv('data/auto_train_data.csv')