数据挖掘实战：特征工程python实战

最新推荐文章于 2023-09-07 00:30:00 发布
最新推荐文章于 2023-09-07 00:30:00 发布
阅读量385 收藏
点赞数
分类专栏： 数据挖掘实战
原文链接：https://heartbeat.fritz.ai/a-practical-guide-to-feature-engineering-in-python-8326e40747c8
版权

特征工程是数据科学和机器学习中的重要技巧，对机器模型性能和EDA（exploratory data analysis）的质量有重要影响。本文介绍几种特征工程技巧

详见 notebook

目录

什么是特征工程

  • 使用领域知识来创造特征，使得机器学习算法发挥作用
  • 从原始数据提取特征以转换为算法需要的格式
  • 需要领域知识、数学和编程技巧

数据集

本文使用贷款违约预测数据和送货平台数据，可直接参考 notebook

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore') # 关闭警告


#load loan datasets
loan_demographics = pd.read_csv('traindemographics.csv')
loan_prev = pd.read_csv('trainprevloans.csv')
loan_perf = pd.read_csv('trainperf.csv')
#load logistics dataset
sendy_data = pd.read_csv('sendy_logistics.csv')

# 贷款统计信息
# customerid (顾客id)
# birthdate (顾客生日)
# bank_account_type （银行账号类型）
# longitude_gps（经度）
# latitude_gps （纬度）
# bank_name_clients (n银行名称)
# bank_branch_clients (支行，非必填，所以很多缺失值)
# employment_status_clients (职业状态)
# level_of_education_clients (教育程度)
loan_demographics.head().append(loan_demographics.tail())
customeridbirthdatebank_account_typelongitude_gpslatitude_gpsbank_name_clientsbank_branch_clientsemployment_status_clientslevel_of_education_clients
08a858e135cb22031015cbafc76964ebd1973-10-10 00:00:00.000000Savings3.3192196.528604GT BankNaNNaNNaN
18a858e275c7ea5ec015c82482d7c39961986-01-21 00:00:00.000000Savings3.3255987.119403Sterling BankNaNPermanentNaN
28a858e5b5bd99460015bdc95cd4856341987-04-01 00:00:00.000000Savings5.7461005.563174Fidelity BankNaNNaNNaN
38a858efd5ca70688015cabd1f1e94b551991-07-19 00:00:00.000000Savings3.3628506.642485GT BankNaNPermanentNaN
48a858e785acd3412015acd48f4920d041982-11-22 00:00:00.000000Savings8.45533211.971410GT BankNaNPermanentNaN
43418a858f155554552501555588ca2b3b401985-12-13 00:00:00.000000Other3.2367537.030168Stanbic IBTCNaNPermanentGraduate
43428a858fc65cf978f4015cf97cee3a02ce1982-07-01 00:00:00.000000Savings7.0137494.875662GT BankNaNNaNNaN
43438a858f4f5b66de3a015b66fc83c619021989-09-26 00:00:00.000000Savings6.2955307.092508GT BankNaNPermanentNaN
43448aaae7a74400b28201441c8b625141501985-09-06 00:00:00.000000Savings3.3542066.539070GT BankHEAD OFFICEPermanentPrimary
43458a85896653e2e18b0153e69c1b90265c1975-06-05 00:00:00.000000Savings6.6610147.472700UBANaNPermanentNaN
# trainperf.csv 顾客连续贷款记录，需根据顾客以前贷款信息和其自身信息预测是否会违约
# customerid (顾客id)
# systemloanid (贷款id. 每个顾客每次贷款有一个id)
# loannumber (需要预测的贷款数)
# approveddate (贷款批准日期)
# creationdate (贷款申请日期)
# loanamount (贷款金额)
# totaldue (总需还款金额，贷款金额+利息+其他费用)
# termdays (贷款期限)
# referredby (担保人id，空则没担保)
# good_bad_flag (good = 按时结清贷款; bad = 未按时结清贷款) - 是要预测的目标值
loan_perf.head().append(loan_perf.tail())
customeridsystemloanidloannumberapproveddatecreationdateloanamounttotalduetermdaysreferredbygood_bad_flag
08a2a81a74ce8c05d014cfb32a0da1049301994762122017-07-25 08:22:56.0000002017-07-25 07:22:47.00000030000.034500.030NaNGood
18a85886e54beabf90154c0a29ae757c030196520422017-07-05 17:04:41.0000002017-07-05 16:04:18.00000015000.017250.030NaNGood
28a8588f35438fe12015444567666018e30196658072017-07-06 14:52:57.0000002017-07-06 13:52:51.00000020000.022250.015NaNGood
38a85890754145ace015429211b513e1630199934332017-07-27 19:00:41.0000002017-07-27 18:00:35.00000010000.011500.015NaNGood
48a858970548359cc015488348198186630196236092017-07-03 23:42:45.0000002017-07-03 22:42:39.00000040000.044000.030NaNGood
43638a858e6d58b0cc520158beeb14b22a5a30200316322017-07-30 09:19:42.0000002017-07-30 08:18:30.00000010000.013000.030NaNBad
43648a858ee85cf400f5015cf44ab1c42d5c30199896722017-07-27 15:35:47.0000002017-07-27 14:35:40.00000010000.013000.030NaNBad
43658a858f365b2547f3015b284597147c9430199557632017-07-25 16:25:57.0000002017-07-25 15:24:47.00000010000.011500.015NaNBad
43668a858f935ca09667015ca0ee3bc63f5130197767922017-07-14 13:50:27.0000002017-07-14 12:50:21.00000010000.013000.0308a858eda5c8863ff015c9dead65807bbBad
43678a858fd458639fcc015868eb14b542ad30196712482017-07-06 21:01:06.0000002017-07-06 20:01:01.00000030000.034500.030NaNBad
# trainprevloans.csv: 本数据及是顾客在此次贷款前的所有贷款记录.
# customerid (顾客id)
# systemloanid (贷款id，每个顾客每次贷款都有)
# loannumber (T需要预测的贷款数)
# approveddate (贷款审批通过日期)
# creationdate (贷款申请日期)
# loanamount (贷款金额)
# totaldue (总需还款金额，贷款金额+利息+其他费用) 
# termdays (贷款期限)
# closeddate (贷款还清日期)
# referredby (担保人id，若空则无担保)
# firstduedate (贷款期限超30天的首次应付款日期)
# firstrepaiddate (实际首次付款日期)
loan_prev.head().append(loan_prev.tail())
customeridsystemloanidloannumberapproveddatecreationdateloanamounttotalduetermdayscloseddatereferredbyfirstduedatefirstrepaiddate
08a2a81a74ce8c05d014cfb32a0da104930168232022016-08-15 18:22:40.0000002016-08-15 17:22:32.00000010000.013000.0302016-09-01 16:06:48.000000NaN2016-09-14 00:00:00.0000002016-09-01 15:51:43.000000
18a2a81a74ce8c05d014cfb32a0da104930188380892017-04-28 18:39:07.0000002017-04-28 17:38:53.00000010000.013000.0302017-05-28 14:44:49.000000NaN2017-05-30 00:00:00.0000002017-05-26 00:00:00.000000
28a2a81a74ce8c05d014cfb32a0da104930183171482017-03-05 10:56:25.0000002017-03-05 09:56:19.00000020000.023800.0302017-04-26 22:18:56.000000NaN2017-04-04 00:00:00.0000002017-04-26 22:03:47.000000
38a8588f35438fe12015444567666018e30186154152017-04-09 18:25:55.0000002017-04-09 17:25:42.00000010000.011500.0152017-04-24 01:35:52.000000NaN2017-04-24 00:00:00.0000002017-04-24 00:48:43.000000
48a85890754145ace015429211b513e1630194175422017-06-17 09:29:57.0000002017-06-17 08:29:50.00000010000.011500.0152017-07-14 21:18:43.000000NaN2017-07-03 00:00:00.0000002017-07-14 21:08:35.000000
181788a858899538ddb8e0153a2b555421fc530161175422016-04-16 13:36:34.0000002016-04-16 12:36:28.00000010000.013000.0302016-05-14 00:04:52.000000NaN2016-05-16 00:00:00.0000002016-05-13 18:05:07.000000
181798a858899538ddb8e0153a2b555421fc530176126792016-11-18 14:26:07.0000002016-11-18 13:25:51.00000030000.034400.0302016-12-13 16:08:57.000000NaN2016-12-19 00:00:00.0000002016-12-13 15:53:48.000000
181808a858899538ddb8e0153a2b555421fc530163165342016-06-12 15:30:56.0000002016-06-12 14:30:50.00000010000.013000.0302016-07-09 15:39:00.000000NaN2016-07-12 00:00:00.0000002016-07-09 15:23:56.000000
181818a858f0656b7820c0156c92ca3ba436f30169769112016-08-27 20:03:45.0000002016-08-27 19:03:34.00000010000.013000.0302016-10-15 10:17:54.000000NaN2016-09-26 00:00:00.0000002016-10-15 10:02:45.000000
181828a858faf5679a838015688de3028143d30171525522016-09-14 23:42:14.0000002016-09-14 22:42:05.00000010000.013000.0302016-09-29 19:51:04.000000NaN2016-10-14 00:00:00.0000002016-09-29 19:35:55.000000
# 送货数据
sendy_data.head().append(sendy_data.tail())
Order NoUser IdVehicle TypePlatform TypePersonal or BusinessPlacement - Day of MonthPlacement - Weekday (Mo = 1)Placement - TimeConfirmation - Day of MonthConfirmation - Weekday (Mo = 1)...Destination LatDestination LongRider IdTime from Pickup to Arrivalspeedmanhattan_disthaversine_distbearingcenter_latitudecenter_longitude
0Order_No_4211User_Id_633Bike3Business959:35:46 AM95...-1.30040636.829741Rider_Id_432745.00.0021600.0179781.930333-2.076903-1.30908036.830056
1Order_No_25375User_Id_2285Bike3Personal12511:16:16 AM125...-1.29500436.814358Rider_Id_8561993.00.0014220.14140611.339849-56.392163-1.32322936.856837
2Order_No_1899User_Id_265Bike3Business30212:39:25 PM302...-1.30092136.828195Rider_Id_155455.00.0030470.0225881.880079-64.183866-1.30460336.835807
3Order_No_9336User_Id_1402Bike3Business1559:25:34 AM155...-1.25714736.795063Rider_Id_8551341.00.0017170.0614874.943458-57.091553-1.26922436.813730
4Order_No_27883User_Id_1737Bike1Personal1319:55:18 AM131...-1.29504136.809817Rider_Id_7701214.00.0018970.0461433.724829148.114398-1.28081936.800968
21196Order_No_8834User_Id_2001Bike3Personal2033:54:38 PM203...-1.27528536.802702Rider_Id_9539.00.1540330.0189681.890335-172.912798-1.26684936.803751
21197Order_No_22892User_Id_1796Bike3Business13610:13:34 AM136...-1.33161936.847976Rider_Id_155770.00.0027010.0474433.731709136.829614-1.31938136.836493
21198Order_No_2831User_Id_2956Bike3Business745:06:16 PM74...-1.25841436.804800Rider_Id_6972953.00.0010310.12033810.756212-73.420487-1.27221636.851167
21199Order_No_6174User_Id_2524Bike1Personal439:31:39 AM43...-1.27920936.794872Rider_Id_3471380.00.0019120.1084749.393189-110.208129-1.26462036.834520
21200Order_No_9836User_Id_718Bike3Business2622:19:47 PM262...-1.32015736.830887Rider_Id_1772128.00.0012050.1136519.026448143.161318-1.28767336.806545

10 rows × 35 columns

这里主要有三种特征：数值、类别、日期特征，以下特征工程没有指定顺序，也不一定适用所有场景，需根据实际情况选择

缺失值处理

缺失值的出现可能是顾客没有提供，或因为错误留空，或因为难以估量. 缺失值可能对模型有重大影响，这里列出几种处理方式。
处理方式依赖特征类型

类别特征缺失值处理方法

有:众数填充、顺序填充（用前面/后面的值填充）、编码填充.
在贷款统计信息 loan_demographics 中，有三个类别特征：bank_branch_clients, employment_status_clients, level_of_education_clients 包含缺失值

## 查看缺失值情况
loan_demographics.isna().sum()

customerid                       0
birthdate                        0
bank_account_type                0
longitude_gps                    0
latitude_gps                     0
bank_name_clients                0
bank_branch_clients           4295
employment_status_clients      648
level_of_education_clients    3759
dtype: int64

## 查看类别分布
loan_demographics['employment_status_clients'].value_counts()

Permanent        3146
Self-Employed     348
Student           142
Unemployed         57
Retired             4
Contract            1
Name: employment_status_clients, dtype: int64

可以看到在职状态中，Permanent 出现的最多，因此可以用这个值填充缺失值

## 众数填充 (Permanent)
loan_demographics['employment_status_clients'] = loan_demographics['employment_status_clients'].fillna(value='Permanent')

loan_demographics.isna().sum()

customerid                       0
birthdate                        0
bank_account_type                0
longitude_gps                    0
latitude_gps                     0
bank_name_clients                0
bank_branch_clients           4295
employment_status_clients        0
level_of_education_clients    3759
dtype: int64

数值特征缺失值处理

1、用均值、众数、中位数填充
2、时序填充（前向/后向）
3、用机器学习模型填充：训练机器学习模型填充缺失值
Sendy logistics 数据集中有两项数值特征：Temperature 和 Precipitation in millimeters，以此为例：

sendy_data.head().append(sendy_data.tail())
Order NoUser IdVehicle TypePlatform TypePersonal or BusinessPlacement - Day of MonthPlacement - Weekday (Mo = 1)Placement - TimeConfirmation - Day of MonthConfirmation - Weekday (Mo = 1)...Arrival at Destination - TimeDistance (KM)TemperaturePrecipitation in millimetersPickup LatPickup LongDestination LatDestination LongRider IdTime from Pickup to Arrival
0Order_No_4211User_Id_633Bike3Business959:35:46 AM95...10:39:55 AM420.4NaN-1.31775536.830370-1.30040636.829741Rider_Id_432745
1Order_No_25375User_Id_2285Bike3Personal12511:16:16 AM125...12:17:22 PM1626.4NaN-1.35145336.899315-1.29500436.814358Rider_Id_8561993
2Order_No_1899User_Id_265Bike3Business30212:39:25 PM302...1:00:38 PM3NaNNaN-1.30828436.843419-1.30092136.828195Rider_Id_155455
3Order_No_9336User_Id_1402Bike3Business1559:25:34 AM155...10:05:27 AM919.2NaN-1.28130136.832396-1.25714736.795063Rider_Id_8551341
4Order_No_27883User_Id_1737Bike1Personal1319:55:18 AM131...10:25:37 AM915.4NaN-1.26659736.792118-1.29504136.809817Rider_Id_7701214
21196Order_No_8834User_Id_2001Bike3Personal2033:54:38 PM203...4:20:17 PM328.6NaN-1.25841436.804800-1.27528536.802702Rider_Id_9539
21197Order_No_22892User_Id_1796Bike3Business13610:13:34 AM136...10:46:17 AM726.0NaN-1.30714336.825009-1.33161936.847976Rider_Id_155770
21198Order_No_2831User_Id_2956Bike3Business745:06:16 PM74...6:40:05 PM2029.2NaN-1.28601836.897534-1.25841436.804800Rider_Id_6972953
21199Order_No_6174User_Id_2524Bike1Personal439:31:39 AM43...10:08:15 AM1315.0NaN-1.25003036.874167-1.27920936.794872Rider_Id_3471380
21200Order_No_9836User_Id_718Bike3Business2622:19:47 PM262...3:17:23 PM1230.9NaN-1.25518936.782203-1.32015736.830887Rider_Id_1772128

10 rows × 29 columns

# 注意：以此只用一种填充方式，众数可能有多个，因此带下标0
mean_df = round(sendy_data['Temperature'].mean())
mode_df = round(sendy_data['Temperature'].mode()[0])
median_df = round(sendy_data['Temperature'].median())

#Fill with mean
print("Filling with mean value of {}".format(mean_df))
sendy_data['Temperature'] = sendy_data['Temperature'].fillna(mean_df)

#Fill with mode
print("Filling with modal value of {}".format(mode_df))
sendy_data['Temperature'] = sendy_data['Temperature'].fillna(mode_df)

#Fill with median
print("Filling with median value of {}".format(median_df))
sendy_data['Temperature'] = sendy_data['Temperature'].fillna(median_df)

Filling with mean value of 23.0
Filling with modal value of 23.0
Filling with median value of 23.0

使用模型填充缺失值

这里对Precipitation in millimeters特征进行模型填充，先找到与目标特征相关的特征值加以利用.
The Seaborn 热图有助于找到相关特征.

plt.figure(figsize = (15,10))
sns.heatmap(sendy_data.corr())

<matplotlib.axes._subplots.AxesSubplot at 0x15c46e85cc0>

在这里插入图片描述

上图看出，大部分特征都与之不相关，这里使用最后三个特征： Destination Lat，Destination Long，Time from Pickup to Arrival

from sklearn.linear_model import LinearRegression

lr = LinearRegression()

to_train = ['Precipitation in millimeters', 'Destination Lat', 'Destination Long', 'Time from Pickup to Arrival']
temp_df = sendy_data[to_train]

#Split dataset with missing values and no missing values as test and train set respectively.
x_train = temp_df[temp_df['Precipitation in millimeters'].notnull()].drop(columns='Precipitation in millimeters')
y_train = temp_df[temp_df['Precipitation in millimeters'].notnull()]['Precipitation in millimeters']
x_test = temp_df[temp_df['Precipitation in millimeters'].isnull()].drop(columns='Precipitation in millimeters')

#Fit a simple linear model to the dataset
lr.fit(x_train, y_train)
pred = lr.predict(x_test)

#print fill values
print(np.round(pred, 5))

#Perform filling
sendy_data['Precipitation in millimeters'][sendy_data['Precipitation in millimeters'].isnull()] = pred

[8.03056 8.18609 7.86142 ... 8.46164 7.45028 8.60717]

也可用sklearn.experimental 模块的IterativeImputer 来自动填充

missing_indx = list(sendy_data['Temperature'][sendy_data['Temperature'].isna()].index)
from sklearn.experimental import enable_iterative_imputer  
from sklearn.impute import IterativeImputer
from sklearn.ensemble import RandomForestRegressor

# 使用带随机森林的IterativeImputer
imp = IterativeImputer(RandomForestRegressor(n_estimators=5), max_iter=5, random_state=1)
to_train = ['Temperature', 'Destination Lat', 'Destination Long','Time from Pickup to Arrival']

#填充
sendy_data[to_train] = pd.DataFrame(imp.fit_transform(sendy_data[to_train]), columns=to_train)

sendy_data['Temperature'][missing_indx].head(10)

2     23.70
8     26.06
14    21.66
15    20.42
16    21.46
32    23.84
42    22.32
46    27.18
49    22.34
53    21.36
Name: Temperature, dtype: float64

类别特征处理

类别特征只取有限的值. 如某地点/事物的热度或者app的评分等级(1,2,3,4,5).
本文中，loan_demographics信息的教育程度level_of_education_clients是类别特征，取值有：Secondary, Graduate, Post-Graduate, Primary.
机器学习算法不能直接作用域类别特征原始形式，因此必须转为数值形式，此过程成为编码
编码有多种方式，具体取决于类别特征类型.

类别特征类型

1、有序类别特征：有自然顺序，如评分等级 (1,2,3,4,5).
2、无序类别特征：无先后之分。例如食物：(大米、意面、通心粉).
有序特征编码：若类别数较少，如教育程度 level_of_education_clients ，可人工编码.

loan_demographics['level_of_education_clients'].unique()

array([nan, 'Secondary', 'Graduate', 'Post-Graduate', 'Primary'],
      dtype=object)

#use a simple map function
map_education = {"Primary" : 1, "Secondary": 2, "Graduate": 3, "Post-Graduate": 4}
    
loan_demographics['level_of_education_clients'] = loan_demographics['level_of_education_clients'].map(map_education)
loan_demographics['level_of_education_clients'].value_counts()

3.0    420
2.0     89
4.0     68
1.0     10
Name: level_of_education_clients, dtype: int64

若需自动编码，可以用categorical_encoders库，有大量编码方式。

!pip install category_encoders

Collecting category_encoders
  Downloading category_encoders-2.1.0-py2.py3-none-any.whl (100 kB)
Requirement already satisfied: numpy>=1.11.3 in c:\anaconda3\envs\mytf\lib\site-packages (from category_encoders) (1.18.1)
Collecting patsy>=0.4.1
  Downloading patsy-0.5.1-py2.py3-none-any.whl (231 kB)
Requirement already satisfied: scipy>=0.19.0 in c:\anaconda3\envs\mytf\lib\site-packages (from category_encoders) (1.4.1)
Requirement already satisfied: scikit-learn>=0.20.0 in c:\anaconda3\envs\mytf\lib\site-packages (from category_encoders) (0.22.2.post1)
Requirement already satisfied: pandas>=0.21.1 in c:\anaconda3\envs\mytf\lib\site-packages (from category_encoders) (1.0.2)
Collecting statsmodels>=0.6.1
  Downloading statsmodels-0.11.1-cp37-none-win_amd64.whl (8.2 MB)
Requirement already satisfied: six in c:\anaconda3\envs\mytf\lib\site-packages (from patsy>=0.4.1->category_encoders) (1.14.0)
Requirement already satisfied: joblib>=0.11 in c:\anaconda3\envs\mytf\lib\site-packages (from scikit-learn>=0.20.0->category_encoders) (0.14.1)
Requirement already satisfied: python-dateutil>=2.6.1 in c:\anaconda3\envs\mytf\lib\site-packages (from pandas>=0.21.1->category_encoders) (2.8.1)
Requirement already satisfied: pytz>=2017.2 in c:\anaconda3\envs\mytf\lib\site-packages (from pandas>=0.21.1->category_encoders) (2019.3)
Installing collected packages: patsy, statsmodels, category-encoders
Successfully installed category-encoders-2.1.0 patsy-0.5.1 statsmodels-0.11.1

有序特征若类别数较多，可用标签编码（Label Encoding），对每个类别标记唯一的数值
这里以特征bank_name_clients和bank_branch_clients 作展示，两者类别数量较多

cat_cols = loan_demographics.select_dtypes(include='object').columns # object 为类别特征
for col in cat_cols:
    print("Number of classes in {}".format(col))
    print(loan_demographics[col].nunique())
    print('--------------------------')

Number of classes in customerid
4334
--------------------------
Number of classes in birthdate
3297
--------------------------
Number of classes in bank_account_type
3
--------------------------
Number of classes in bank_name_clients
18
--------------------------
Number of classes in bank_branch_clients
45
--------------------------
Number of classes in employment_status_clients
6
--------------------------

loan_demographics.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4346 entries, 0 to 4345
Data columns (total 9 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   customerid                  4346 non-null   object 
 1   birthdate                   4346 non-null   object 
 2   bank_account_type           4346 non-null   object 
 3   longitude_gps               4346 non-null   float64
 4   latitude_gps                4346 non-null   float64
 5   bank_name_clients           4346 non-null   object 
 6   bank_branch_clients         51 non-null     object 
 7   employment_status_clients   4346 non-null   object 
 8   level_of_education_clients  587 non-null    float64
dtypes: float64(3), object(6)
memory usage: 305.7+ KB

categorical_encoders 库的OrdinalEncoder 函数可用于编码

import category_encoders as ce

#Label encoding
cat_cols = ['bank_name_clients', 'bank_branch_clients']
encoder = ce.OrdinalEncoder(cols=cat_cols)
loan_demographics = encoder.fit_transform(loan_demographics)

loan_demographics.bank_name_clients.value_counts()

1     1598
11     590
4      432
8      353
9      301
10     282
5      150
13     145
6      121
3      103
7      102
2       55
18      28
12      23
15      22
16      19
14      15
17       7
Name: bank_name_clients, dtype: int64

独特编码

使用二进制值表示类别，每种类别创建一个特征，在类别特征可取值变多时，效率降低。

cats = ['bank_account_type','level_of_education_clients']
one_hot_enc = ce.OneHotEncoder(cols=cats)
loan_demographics = one_hot_enc.fit_transform(loan_demographics)
loan_demographics.head().T
01234
customerid8a858e135cb22031015cbafc76964ebd8a858e275c7ea5ec015c82482d7c39968a858e5b5bd99460015bdc95cd4856348a858efd5ca70688015cabd1f1e94b558a858e785acd3412015acd48f4920d04
birthdate1973-10-10 00:00:00.0000001986-01-21 00:00:00.0000001987-04-01 00:00:00.0000001991-07-19 00:00:00.0000001982-11-22 00:00:00.000000
bank_account_type_111111
bank_account_type_200000
bank_account_type_300000
longitude_gps3.319223.32565.74613.362858.45533
latitude_gps6.52867.11945.563176.6424911.9714
bank_name_clients12311
bank_branch_clients11111
employment_status_clientsPermanentPermanentPermanentPermanentPermanent
level_of_education_clients_111111
level_of_education_clients_200000
level_of_education_clients_300000
level_of_education_clients_400000
level_of_education_clients_500000

哈希编码

是一种快速、节省空间的特征编码方式. 在类别特征取值多时比较高效。对特征值施加hash函数

cat_cols = ['bank_name_clients', 'bank_branch_clients']
hash_enc = ce.HashingEncoder(cols=cat_cols, n_components=10)
loan_demographics = hash_enc.fit_transform(loan_demographics)
loan_demographics.head()
customeridbirthdatebank_account_typelongitude_gpslatitude_gpsbank_name_clientsbank_branch_clientsemployment_status_clientslevel_of_education_clients
08a858e135cb22031015cbafc76964ebd1973-10-10 00:00:00.000000Savings3.3192196.52860411PermanentNaN
18a858e275c7ea5ec015c82482d7c39961986-01-21 00:00:00.000000Savings3.3255987.11940321PermanentNaN
28a858e5b5bd99460015bdc95cd4856341987-04-01 00:00:00.000000Savings5.7461005.56317431PermanentNaN
38a858efd5ca70688015cabd1f1e94b551991-07-19 00:00:00.000000Savings3.3628506.64248511PermanentNaN
48a858e785acd3412015acd48f4920d041982-11-22 00:00:00.000000Savings8.45533211.97141011PermanentNaN

更多编码详见：https://contrib.scikit-learn.org/categorical-encoding/

数值/连续特征的处理

数值/连续特征是数据集中最常见特征的形式。关于此类特征的工程往往基于领域知识，可做以下处理:
对数变换：中心化 （正则分布），可让大部分算法效果更好.
注意：当对目标特征做对数变换时，在做模型解释时需要加上指数变换。
对数变换常用于有偏特征，或左偏或有偏，偏度可用图像观察。
这里对sendy数据集的Distance 特征做分析，为右偏

sns.distplot(sendy_data['Distance (KM)'])
plt.title("Histogram of Distance (KM)")
plt.show()

在这里插入图片描述

sendy_data['Distance (KM)'] =  np.log1p(sendy_data['Distance (KM)']) # 使用加1后去对数，避免对0取对数
sns.distplot(sendy_data['Distance (KM)'])
plt.title("Log-Transformed plot of Distance (KM)")
plt.show()

在这里插入图片描述

使用领域知识构造特征

当对数据的领域了解时，可以依此创建一些对模型有帮助的特征，示例如下：
1、利息 Interest elapsed：为总需还款金额totaldue 与贷款金额loanamount之差

loan_prev['interest_elapsed'] = loan_prev['totaldue'] - loan_prev['loanamount']

loan_prev['interest_elapsed']

0        3000.0
1        3000.0
2        3800.0
3        1500.0
4        1500.0
          ...  
18178    3000.0
18179    4400.0
18180    3000.0
18181    3000.0
18182    3000.0
Name: interest_elapsed, Length: 18183, dtype: float64

loan_prev
customeridsystemloanidloannumberapproveddatecreationdateloanamounttotalduetermdayscloseddatereferredbyfirstduedatefirstrepaiddateinterest_elapsed
08a2a81a74ce8c05d014cfb32a0da104930168232022016-08-15 18:22:40.0000002016-08-15 17:22:32.00000010000.013000.0302016-09-01 16:06:48.000000NaN2016-09-14 00:00:00.0000002016-09-01 15:51:43.0000003000.0
18a2a81a74ce8c05d014cfb32a0da104930188380892017-04-28 18:39:07.0000002017-04-28 17:38:53.00000010000.013000.0302017-05-28 14:44:49.000000NaN2017-05-30 00:00:00.0000002017-05-26 00:00:00.0000003000.0
28a2a81a74ce8c05d014cfb32a0da104930183171482017-03-05 10:56:25.0000002017-03-05 09:56:19.00000020000.023800.0302017-04-26 22:18:56.000000NaN2017-04-04 00:00:00.0000002017-04-26 22:03:47.0000003800.0
38a8588f35438fe12015444567666018e30186154152017-04-09 18:25:55.0000002017-04-09 17:25:42.00000010000.011500.0152017-04-24 01:35:52.000000NaN2017-04-24 00:00:00.0000002017-04-24 00:48:43.0000001500.0
48a85890754145ace015429211b513e1630194175422017-06-17 09:29:57.0000002017-06-17 08:29:50.00000010000.011500.0152017-07-14 21:18:43.000000NaN2017-07-03 00:00:00.0000002017-07-14 21:08:35.0000001500.0
..........................................
181788a858899538ddb8e0153a2b555421fc530161175422016-04-16 13:36:34.0000002016-04-16 12:36:28.00000010000.013000.0302016-05-14 00:04:52.000000NaN2016-05-16 00:00:00.0000002016-05-13 18:05:07.0000003000.0
181798a858899538ddb8e0153a2b555421fc530176126792016-11-18 14:26:07.0000002016-11-18 13:25:51.00000030000.034400.0302016-12-13 16:08:57.000000NaN2016-12-19 00:00:00.0000002016-12-13 15:53:48.0000004400.0
181808a858899538ddb8e0153a2b555421fc530163165342016-06-12 15:30:56.0000002016-06-12 14:30:50.00000010000.013000.0302016-07-09 15:39:00.000000NaN2016-07-12 00:00:00.0000002016-07-09 15:23:56.0000003000.0
181818a858f0656b7820c0156c92ca3ba436f30169769112016-08-27 20:03:45.0000002016-08-27 19:03:34.00000010000.013000.0302016-10-15 10:17:54.000000NaN2016-09-26 00:00:00.0000002016-10-15 10:02:45.0000003000.0
181828a858faf5679a838015688de3028143d30171525522016-09-14 23:42:14.0000002016-09-14 22:42:05.00000010000.013000.0302016-09-29 19:51:04.000000NaN2016-10-14 00:00:00.0000002016-09-29 19:35:55.0000003000.0

18183 rows × 13 columns

2、贷款数量Loan count：根据顾客id分组，累计loannumber

loan_prev.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 18183 entries, 0 to 18182
Data columns (total 13 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   customerid        18183 non-null  object 
 1   systemloanid      18183 non-null  int64  
 2   loannumber        18183 non-null  int64  
 3   approveddate      18183 non-null  object 
 4   creationdate      18183 non-null  object 
 5   loanamount        18183 non-null  float64
 6   totaldue          18183 non-null  float64
 7   termdays          18183 non-null  int64  
 8   closeddate        18183 non-null  object 
 9   referredby        1026 non-null   object 
 10  firstduedate      18183 non-null  object 
 11  firstrepaiddate   18183 non-null  object 
 12  interest_elapsed  18183 non-null  float64
dtypes: float64(3), int64(3), object(7)
memory usage: 1.8+ MB

#Groupby customer id and calculate their total loans taken
loannumber_count = loan_prev.groupby(by='customerid').agg(['count'])[['loannumber']].reset_index()

#merge back to dataset on customer_id
loan_prev = loan_prev.merge(right=loannumber_count, how='left', on='customerid')
loan_prev.head()
customeridsystemloanidloannumberapproveddatecreationdateloanamounttotalduetermdayscloseddatereferredbyfirstduedatefirstrepaiddate(loannumber, count)
08a2a81a74ce8c05d014cfb32a0da104930168232022016-08-15 18:22:40.0000002016-08-15 17:22:32.00000010000.013000.0302016-09-01 16:06:48.000000NaN2016-09-14 00:00:00.0000002016-09-01 15:51:43.00000011
18a2a81a74ce8c05d014cfb32a0da104930188380892017-04-28 18:39:07.0000002017-04-28 17:38:53.00000010000.013000.0302017-05-28 14:44:49.000000NaN2017-05-30 00:00:00.0000002017-05-26 00:00:00.00000011
28a2a81a74ce8c05d014cfb32a0da104930183171482017-03-05 10:56:25.0000002017-03-05 09:56:19.00000020000.023800.0302017-04-26 22:18:56.000000NaN2017-04-04 00:00:00.0000002017-04-26 22:03:47.00000011
38a8588f35438fe12015444567666018e30186154152017-04-09 18:25:55.0000002017-04-09 17:25:42.00000010000.011500.0152017-04-24 01:35:52.000000NaN2017-04-24 00:00:00.0000002017-04-24 00:48:43.0000006
48a85890754145ace015429211b513e1630194175422017-06-17 09:29:57.0000002017-06-17 08:29:50.00000010000.011500.0152017-07-14 21:18:43.000000NaN2017-07-03 00:00:00.0000002017-07-14 21:08:35.0000002
loan_prev.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 18183 entries, 0 to 18182
Data columns (total 13 columns):
 #   Column               Non-Null Count  Dtype  
---  ------               --------------  -----  
 0   customerid           18183 non-null  object 
 1   systemloanid         18183 non-null  int64  
 2   loannumber           18183 non-null  int64  
 3   approveddate         18183 non-null  object 
 4   creationdate         18183 non-null  object 
 5   loanamount           18183 non-null  float64
 6   totaldue             18183 non-null  float64
 7   termdays             18183 non-null  int64  
 8   closeddate           18183 non-null  object 
 9   referredby           1026 non-null   object 
 10  firstduedate         18183 non-null  object 
 11  firstrepaiddate      18183 non-null  object 
 12  (loannumber, count)  18183 non-null  int64  
dtypes: float64(2), int64(4), object(7)
memory usage: 1.9+ MB

3、速度Speed：发货距离Distance (KM)与时间Time from Pickup to Arrival之比

#create feature speed in sendy dataset
sendy_data['speed'] = sendy_data['Distance (KM)'] / sendy_data['Time from Pickup to Arrival']
sendy_data.head().T
01234
Order NoOrder_No_4211Order_No_25375Order_No_1899Order_No_9336Order_No_27883
User IdUser_Id_633User_Id_2285User_Id_265User_Id_1402User_Id_1737
Vehicle TypeBikeBikeBikeBikeBike
Platform Type33331
Personal or BusinessBusinessPersonalBusinessBusinessPersonal
Placement - Day of Month912301513
Placement - Weekday (Mo = 1)55251
Placement - Time9:35:46 AM11:16:16 AM12:39:25 PM9:25:34 AM9:55:18 AM
Confirmation - Day of Month912301513
Confirmation - Weekday (Mo = 1)55251
Confirmation - Time9:40:10 AM11:23:21 AM12:42:44 PM9:26:05 AM9:56:18 AM
Arrival at Pickup - Day of Month912301513
Arrival at Pickup - Weekday (Mo = 1)55251
Arrival at Pickup - Time10:04:47 AM11:40:22 AM12:49:34 PM9:37:56 AM10:03:53 AM
Pickup - Day of Month912301513
Pickup - Weekday (Mo = 1)55251
Pickup - Time10:27:30 AM11:44:09 AM12:53:03 PM9:43:06 AM10:05:23 AM
Arrival at Destination - Day of Month912301513
Arrival at Destination - Weekday (Mo = 1)55251
Arrival at Destination - Time10:39:55 AM12:17:22 PM1:00:38 PM10:05:27 AM10:25:37 AM
Distance (KM)1.609442.833211.386292.302592.30259
Temperature20.426.423.719.215.4
Precipitation in millimetersNaNNaNNaNNaNNaN
Pickup Lat-1.31775-1.35145-1.30828-1.2813-1.2666
Pickup Long36.830436.899336.843436.832436.7921
Destination Lat-1.30041-1.295-1.30092-1.25715-1.29504
Destination Long36.829736.814436.828236.795136.8098
Rider IdRider_Id_432Rider_Id_856Rider_Id_155Rider_Id_855Rider_Id_770
Time from Pickup to Arrival745199345513411214
speed0.002160320.001421580.00304680.001717070.00189669

多项式（交叉）特征

多项式特征是在特征之间创建关联，有助于捕捉特征之间的关系，减少模型偏差（只要不过拟合）.
可以通过特征之间的加、乘、除来创建交叉特征.
这里用sklearn的特征模块，从loan previous 数据集的loannumber, totaldue,termdays 创建特征

#Use Sklearn Polynomial Features
from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures()
to_cross = ['loannumber', 'totaldue', 'termdays']
crossed_feats = poly.fit_transform(loan_prev[to_cross].values)

#Convert to Pandas DataFrame and merge to original dataset
crossed_feats = pd.DataFrame(crossed_feats)
loan_prev = pd.concat([loan_prev, crossed_feats], axis=1)

loan_prev.head().T
01234
customerid8a2a81a74ce8c05d014cfb32a0da10498a2a81a74ce8c05d014cfb32a0da10498a2a81a74ce8c05d014cfb32a0da10498a8588f35438fe12015444567666018e8a85890754145ace015429211b513e16
systemloanid301682320301883808301831714301861541301941754
loannumber29852
approveddate2016-08-15 18:22:40.0000002017-04-28 18:39:07.0000002017-03-05 10:56:25.0000002017-04-09 18:25:55.0000002017-06-17 09:29:57.000000
creationdate2016-08-15 17:22:32.0000002017-04-28 17:38:53.0000002017-03-05 09:56:19.0000002017-04-09 17:25:42.0000002017-06-17 08:29:50.000000
loanamount1000010000200001000010000
totaldue1300013000238001150011500
termdays3030301515
closeddate2016-09-01 16:06:48.0000002017-05-28 14:44:49.0000002017-04-26 22:18:56.0000002017-04-24 01:35:52.0000002017-07-14 21:18:43.000000
referredbyNaNNaNNaNNaNNaN
firstduedate2016-09-14 00:00:00.0000002017-05-30 00:00:00.0000002017-04-04 00:00:00.0000002017-04-24 00:00:00.0000002017-07-03 00:00:00.000000
firstrepaiddate2016-09-01 15:51:43.0000002017-05-26 00:00:00.0000002017-04-26 22:03:47.0000002017-04-24 00:48:43.0000002017-07-14 21:08:35.000000
(loannumber, count)11111162
011111
129852
21300013000238001150011500
33030301515
448164254
5260001170001904005750023000
6602702407530
71.69e+081.69e+085.6644e+081.3225e+081.3225e+08
8390000390000714000172500172500
9900900900225225

特征标准化

标准化将数值特征分布修改到正常范围，不破坏数据取值范围和信息。
正则化对于基于距离的模型如KNN十分重要，也有助于提高神经网络的训练速度.
sklearn 提供的标准化函数:
StandardScaler: 减去均值、缩放到单位方差.
RobustScaler: 使用对异常值有鲁棒性的统计信息来缩放特征
MinMaxScaler: 将特征缩放到指定范围（范围可自定义）.
注意: 不要将标准化函数在测试/验证集上训练。
sklearn的 标准化函数对缺失值不具有鲁棒性，因此使用前要做缺失值填充.

from sklearn.preprocessing import StandardScaler

feats = ['loannumber', 'totaldue', 'termdays']
sc = StandardScaler()
sc_data = sc.fit_transform(loan_prev[feats])
sc_data

array([[-0.67377132, -0.62877649,  0.30213166],
       [ 1.48047238, -0.62877649,  0.30213166],
       [ 1.17272328,  0.40432506,  0.30213166],
       ...,
       [-0.05827312, -0.62877649,  0.30213166],
       [-0.98152042, -0.62877649,  0.30213166],
       [-0.67377132, -0.62877649,  0.30213166]])

sc_data.shape

(18183, 3)

from sklearn.preprocessing import RobustScaler
robsc = RobustScaler()

rb_data = robsc.fit_transform(loan_prev[feats])
rb_data

array([[-0.25      ,  0.        ,  0.        ],
       [ 1.5       ,  0.        ,  0.        ],
       [ 1.25      ,  0.83076923,  0.        ],
       ...,
       [ 0.25      ,  0.        ,  0.        ],
       [-0.5       ,  0.        ,  0.        ],
       [-0.25      ,  0.        ,  0.        ]])

from sklearn.preprocessing import MinMaxScaler

minsc = MinMaxScaler(feature_range=(0,2))
minmax_data = minsc.fit_transform(loan_prev[feats])
minmax_data

array([[0.08      , 0.29543697, 0.4       ],
       [0.64      , 0.29543697, 0.4       ],
       [0.56      , 0.6295437 , 0.4       ],
       ...,
       [0.24      , 0.29543697, 0.4       ],
       [0.        , 0.29543697, 0.4       ],
       [0.08      , 0.29543697, 0.4       ]])

日期特征处理

对时间特征可做很多处理，如时间差
对loan performance 数据集的申请时间creationdate 和审批通过时间做处理

#First convert to pandas datetime format
loan_perf['approveddate'] = pd.to_datetime(loan_perf['approveddate'])
loan_perf['creationdate'] = pd.to_datetime(loan_perf['creationdate'])
loan_perf['date_elapsed_in_secs'] = (loan_perf['approveddate'] - loan_perf['creationdate']) / np.timedelta64(1,'s') #can subtitute with [h,m,s]
loan_perf.head()
customeridsystemloanidloannumberapproveddatecreationdateloanamounttotalduetermdaysreferredbygood_bad_flagdate_elapsed_in_secs
08a2a81a74ce8c05d014cfb32a0da1049301994762122017-07-25 08:22:562017-07-25 07:22:4730000.034500.030NaN03609.0
18a85886e54beabf90154c0a29ae757c030196520422017-07-05 17:04:412017-07-05 16:04:1815000.017250.030NaN03623.0
28a8588f35438fe12015444567666018e30196658072017-07-06 14:52:572017-07-06 13:52:5120000.022250.015NaN03606.0
38a85890754145ace015429211b513e1630199934332017-07-27 19:00:412017-07-27 18:00:3510000.011500.015NaN03606.0
48a858970548359cc015488348198186630196236092017-07-03 23:42:452017-07-03 22:42:3940000.044000.030NaN03606.0

以下展示如何提取时间特征如：天、周、小时、秒等:

#First convert to pandas datetime format
loan_perf['approveddate'] = pd.to_datetime(loan_perf['approveddate'])

#use pandas built in functions
loan_perf['approved_day'] = loan_perf['approveddate'].dt.day
loan_perf['approved_week'] = loan_perf['approveddate'].dt.week
loan_perf['approved_hour'] = loan_perf['approveddate'].dt.hour

Pandas date特征还有其他属性可尝试.
还可以提取一天中的阶段(morning, afternoon, evenings)

def map_hours(x):
    if x in [0,1,2,3,4,5,6,7,8,9,10,11,12]:
        return 'morning'
    elif x in [13,14,15,16]:
        return 'afternoon'
    else:
        return 'evening'
    
loan_perf['period_of_day'] = loan_perf['approved_hour'].map(map_hours)
loan_perf.head()
customeridsystemloanidloannumberapproveddatecreationdateloanamounttotalduetermdaysreferredbygood_bad_flagdate_elapsed_in_secsapproved_dayapproved_weekapproved_hourperiod_of_day
08a2a81a74ce8c05d014cfb32a0da1049301994762122017-07-25 08:22:562017-07-25 07:22:4730000.034500.030NaN03609.025308morning
18a85886e54beabf90154c0a29ae757c030196520422017-07-05 17:04:412017-07-05 16:04:1815000.017250.030NaN03623.052717evening
28a8588f35438fe12015444567666018e30196658072017-07-06 14:52:572017-07-06 13:52:5120000.022250.015NaN03606.062714afternoon
38a85890754145ace015429211b513e1630199934332017-07-27 19:00:412017-07-27 18:00:3510000.011500.015NaN03606.0273019evening
48a858970548359cc015488348198186630196236092017-07-03 23:42:452017-07-03 22:42:3940000.044000.030NaN03606.032723evening

datasist库还有更多时间处理函数，详见：https://towardsdatascience.com/https-medium-com-risingdeveloper-easy-data-analysis-visualization-and-modeling-using-datasist-part1-8b26526dbe01

地理位置特征处理

数据集中的经纬度，地址都是地理位置特征.
对于经纬度特征可以做很多处理，可以用Geojson或 Geopy的库将位置特征值转为地图上的地址.
但这类方法很慢，本文展示更简单且更快的方法
以下示例来自于Kaggle.
曼哈顿距离: 两点间的水平距离和垂直距离之和

#曼哈顿距离
def manhattan_distance(lat1, lng1, lat2, lng2):
    a = np.abs(lat2 -lat1)
    b = np.abs(lng1 - lng2)
    return a + b
sendy_data['manhattan_dist'] = manhattan_distance(sendy_data['Pickup Lat'].values, sendy_data['Pickup Long'].values,
                                               sendy_data['Destination Lat'].values, sendy_data['Destination Long'].values)
sendy_data.head()
Order NoUser IdVehicle TypePlatform TypePersonal or BusinessPlacement - Day of MonthPlacement - Weekday (Mo = 1)Placement - TimeConfirmation - Day of MonthConfirmation - Weekday (Mo = 1)...TemperaturePrecipitation in millimetersPickup LatPickup LongDestination LatDestination LongRider IdTime from Pickup to Arrivalspeedmanhattan_dist
0Order_No_4211User_Id_633Bike3Business959:35:46 AM95...20.4NaN-1.31775536.830370-1.30040636.829741Rider_Id_432745.00.0021600.017978
1Order_No_25375User_Id_2285Bike3Personal12511:16:16 AM125...26.4NaN-1.35145336.899315-1.29500436.814358Rider_Id_8561993.00.0014220.141406
2Order_No_1899User_Id_265Bike3Business30212:39:25 PM302...23.7NaN-1.30828436.843419-1.30092136.828195Rider_Id_155455.00.0030470.022588
3Order_No_9336User_Id_1402Bike3Business1559:25:34 AM155...19.2NaN-1.28130136.832396-1.25714736.795063Rider_Id_8551341.00.0017170.061487
4Order_No_27883User_Id_1737Bike1Personal1319:55:18 AM131...15.4NaN-1.26659736.792118-1.29504136.809817Rider_Id_7701214.00.0018970.046143

5 rows × 31 columns

Haversine距离:是两点在球体上的距离, 在导航中很重要.

#Haversine distance
def haversine_array(lat1, lng1, lat2, lng2):
    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))
    AVG_EARTH_RADIUS = 6371  # in km
    lat = lat2 - lat1
    lng = lng2 - lng1
    d = np.sin(lat * 0.5) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin(lng * 0.5) ** 2
    h = 2 * AVG_EARTH_RADIUS * np.arcsin(np.sqrt(d))
    return h
sendy_data['haversine_dist'] = haversine_array(sendy_data['Pickup Lat'].values, sendy_data['Pickup Long'].values,
                                                   sendy_data['Destination Lat'].values, sendy_data['Destination Long'].values)
sendy_data.head()
Order NoUser IdVehicle TypePlatform TypePersonal or BusinessPlacement - Day of MonthPlacement - Weekday (Mo = 1)Placement - TimeConfirmation - Day of MonthConfirmation - Weekday (Mo = 1)...Precipitation in millimetersPickup LatPickup LongDestination LatDestination LongRider IdTime from Pickup to Arrivalspeedmanhattan_disthaversine_dist
0Order_No_4211User_Id_633Bike3Business959:35:46 AM95...NaN-1.31775536.830370-1.30040636.829741Rider_Id_432745.00.0021600.0179781.930333
1Order_No_25375User_Id_2285Bike3Personal12511:16:16 AM125...NaN-1.35145336.899315-1.29500436.814358Rider_Id_8561993.00.0014220.14140611.339849
2Order_No_1899User_Id_265Bike3Business30212:39:25 PM302...NaN-1.30828436.843419-1.30092136.828195Rider_Id_155455.00.0030470.0225881.880079
3Order_No_9336User_Id_1402Bike3Business1559:25:34 AM155...NaN-1.28130136.832396-1.25714736.795063Rider_Id_8551341.00.0017170.0614874.943458
4Order_No_27883User_Id_1737Bike1Personal1319:55:18 AM131...NaN-1.26659736.792118-1.29504136.809817Rider_Id_7701214.00.0018970.0461433.724829

5 rows × 32 columns

  1. Bearing角度: 是从起始点到目的地的转向角度, 必须在 0 到 360之间.
#Bearing
def bearing_array(lat1, lng1, lat2, lng2):
    AVG_EARTH_RADIUS = 6371  # in km
    lng_delta_rad = np.radians(lng2 - lng1)
    lat1, lng1, lat2, lng2 = map(np.radians, (lat1, lng1, lat2, lng2))
    y = np.sin(lng_delta_rad) * np.cos(lat2)
    x = np.cos(lat1) * np.sin(lat2) - np.sin(lat1) * np.cos(lat2) * np.cos(lng_delta_rad)
    return np.degrees(np.arctan2(y, x))
sendy_data['bearing'] = bearing_array(sendy_data['Pickup Lat'].values, sendy_data['Pickup Long'].values,
                                                   sendy_data['Destination Lat'].values, sendy_data['Destination Long'].values)
sendy_data.head()
Order NoUser IdVehicle TypePlatform TypePersonal or BusinessPlacement - Day of MonthPlacement - Weekday (Mo = 1)Placement - TimeConfirmation - Day of MonthConfirmation - Weekday (Mo = 1)...Pickup LatPickup LongDestination LatDestination LongRider IdTime from Pickup to Arrivalspeedmanhattan_disthaversine_distbearing
0Order_No_4211User_Id_633Bike3Business959:35:46 AM95...-1.31775536.830370-1.30040636.829741Rider_Id_432745.00.0021600.0179781.930333-2.076903
1Order_No_25375User_Id_2285Bike3Personal12511:16:16 AM125...-1.35145336.899315-1.29500436.814358Rider_Id_8561993.00.0014220.14140611.339849-56.392163
2Order_No_1899User_Id_265Bike3Business30212:39:25 PM302...-1.30828436.843419-1.30092136.828195Rider_Id_155455.00.0030470.0225881.880079-64.183866
3Order_No_9336User_Id_1402Bike3Business1559:25:34 AM155...-1.28130136.832396-1.25714736.795063Rider_Id_8551341.00.0017170.0614874.943458-57.091553
4Order_No_27883User_Id_1737Bike1Personal1319:55:18 AM131...-1.26659736.792118-1.29504136.809817Rider_Id_7701214.00.0018970.0461433.724829148.114398

5 rows × 33 columns

中心点:计算两点经纬度之间的中点

#Get center of lat and longitude
sendy_data['center_latitude'] = (sendy_data['Pickup Lat'].values + sendy_data['Destination Lat'].values) / 2
sendy_data['center_longitude'] = (sendy_data['Pickup Long'].values + sendy_data['Destination Long'].values) / 2
sendy_data.head()
Order NoUser IdVehicle TypePlatform TypePersonal or BusinessPlacement - Day of MonthPlacement - Weekday (Mo = 1)Placement - TimeConfirmation - Day of MonthConfirmation - Weekday (Mo = 1)...Destination LatDestination LongRider IdTime from Pickup to Arrivalspeedmanhattan_disthaversine_distbearingcenter_latitudecenter_longitude
0Order_No_4211User_Id_633Bike3Business959:35:46 AM95...-1.30040636.829741Rider_Id_432745.00.0021600.0179781.930333-2.076903-1.30908036.830056
1Order_No_25375User_Id_2285Bike3Personal12511:16:16 AM125...-1.29500436.814358Rider_Id_8561993.00.0014220.14140611.339849-56.392163-1.32322936.856837
2Order_No_1899User_Id_265Bike3Business30212:39:25 PM302...-1.30092136.828195Rider_Id_155455.00.0030470.0225881.880079-64.183866-1.30460336.835807
3Order_No_9336User_Id_1402Bike3Business1559:25:34 AM155...-1.25714736.795063Rider_Id_8551341.00.0017170.0614874.943458-57.091553-1.26922436.813730
4Order_No_27883User_Id_1737Bike1Personal1319:55:18 AM131...-1.29504136.809817Rider_Id_7701214.00.0018970.0461433.724829148.114398-1.28081936.800968

5 rows × 35 columns

datasist库有更多用法

空腹熊猫
关注
专栏目录
特征工程实战泰坦尼克号
qq_45531594的博客
09-02 396
数据标签介绍 pclass ： 船舱等级，一等舱、二等舱 sibsp ： 有无堂兄弟 parch ：带没带父母小孩 import pandas as pd import numpy as np import warnings warnings.filterwarnings("ignore") df_train = pd.read_csv('C:/Users/lb/Desktop/test/train.csv') #默认看前五行数据 df_train.head() df_train.inf
机器学习实战——特征工程之数据预处理
Yucen的博客
09-26 873
机器学习实战的特征工程主要包含数据预处理、特征构建、特征选择三步，首先来介绍数据预处理。 我选择python作为工具，并将主要用到pandas、numpy等数据工具库。加载库： import pandas as pd import numpy as np 1. 观察数据 （1）查看数据的行数和列数 print("TRAIN shape:",train.shape) （2）查看数据的...
特征工程实战篇
[ 25'h ]
07-30 1146
  当然，如果能获取更多的实际业务背景知识，则能够进行更加深入的数据分析与用户挽留策略的制定。不过需要知道的是，无论是作为实际建模预测项目，还是结合实际业务进行数据分析，在完成数据清洗后对变量进行相关性分析，都是了解数据情况的重要手段，也是所有建模过程中必备的环节。在后续的内容中，我们也将在此基础上进一步来进行特征工程以及模型训练的相关工作，最终借助模型，来进行实时的用户流失预测，并且根据最终的模型结果来更精确的判别变量重要性，以及根据模型方程来判断变量影响流失概率的量化结果。.....
特征工程-特征选择实战
m0_52118763的博客
01-20 2173
特征选择 1.特征选择介绍 以鸢尾花为例，如何从以下特征中选择出合适的特征。 可能很多人都知道和前四个特征有关，Color和Time这两个特征是被舍弃的 但是如何判断前四个特征就是我们最需要的，这就是特征选择 特征选择:对坏属性说不 选择特征是为了提高预测能力，降低时间成本 特征选择的方法: 基于统计的特征选择 基于模型的特征选择 基于统计的特征选择(都是单变量方法) 皮尔逊相关系数(Pearson correlations) 假设检验 2 .数据导入及探索 判断信用卡逾期 选择特征是为了提
python数据分析及特征工程（实战）
CarryLvan的博客
06-18 5369
python数据分析及特征工程（实战）1.数据分析1.1单属性分析1.2多属性分析2.特征工程2.1 数据清洗2.2 特征选择2.3 特征构造2.4 特征降维 1.数据分析 1.1单属性分析 1.2多属性分析 2.特征工程 2.1 数据清洗 2.2 特征选择 2.3 特征构造 2.4 特征降维 ...
Python数据分析与挖掘实战.zip_Python 数据挖掘_python_python 案例_python数据分析_挖掘实战
07-14
Python的数据分析和挖掘的案例，非常不错
数据分析实战：利用python对心脏病数据集进行分析
01-06
↑ 关注 + 星标 ~ 有趣的不像个技术号 每晚九点，我们准时相约  我们都很害怕生病，但感冒发烧这种从小到大的疾病我们已经麻木了，...今天在kaggle上看到一个心脏病数据（数据集下载地址和源码见文末），那么借此
Python数据分析与挖掘实战_Python数据分析与挖掘实战_python_数据分析_
09-30
在数据挖掘部分，我们可以使用Scikit-learn进行特征工程，包括特征选择、特征缩放、特征构造等。之后，根据问题类型选择合适的模型，如线性回归、决策树、随机森林、支持向量机等，并通过训练集进行训练。模型评估则...
Python数据处理实战：基于真实场景的数据
06-22
 Python数据处理实战: 基于真实场景的数据(Python数据处理和特征工程)作为Python数据清洗实战入门课程的升级版，本课程以真实的场景数据为案例进行教学，包括征信，电商，零售数据等, 本课程由浅入深详细讲解Python...
Python3数据分析与挖掘实战:源代码code
最新发布
05-02
Python3数据分析与挖掘实战:源代码code k_means.py jcs. py BitVector.py bloom.py lesson.py aprio ri.py wenben.py phantomjs.py 第9周 课外拓展学习资 料 第6周 第7周 第8周 第1周 第2周 第3周 第4周 第5周 第10...
干货分享 | 详解特征工程与推荐系统及其实践（附PPT）
量子位
11-19 1110
云脑科技机器学习训练营11月13日正式开始啦！量子位作为合作媒体独家为大家分享课程干货内容。 本期内容简介 主题：亿级用户电商平台推荐系统挑战 主讲人：张本宇（云脑科技创始人&CEO） 从事人工智能十八年的老兵，曾就职于微软亚洲研究院、Google和Facebook，在AI方面手握了150项美国专利，在国际一流期刊及会议上发表的45篇论文已被引用超过6000次，创新工场最新
数据分析-特征工程入门实战
u013010473的博客
04-06 317
文章目录Step 1:导入函数工具箱Step 2:数据读取Step3:特征与标签构建Step 4:模型训练与预测 Step 1:导入函数工具箱 ## 基础工具 import numpy as np import pandas as pd import warnings import matplotlib import matplotlib.pyplot as plt import seaborn...
机器学习实战----特征工程
只布布倩
05-18 1164
一 什么是特征工程       数据和特征决定了机器学习的上限，而模型和算法只是逼近这个上限。其本质是一项工程活动，目的是最大限度的从原始数据中提取特征以供算法和模型使用。什么是“数据和特征决定了机器学习的上限，而模型和算法只是逼近这个上限”？就像我们学习一本书，书中的数据和从书中能够获得的所有知识是这本书的知识上限，而我们通过各种学习方法获得的知识是接近这个上限。数据和特征就像一本书中的知识的上...
Kaggle实战（三）： 数据集的特征工程
小明同学
12-05 965
本节主要介绍对于所给的数据，进行特征变换以及构造新的特征 import numpy as np import pandas as pd from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfVectorizer from scip...
数据挖掘实战（二）：特征工程
weixin_40460554的博客
08-08 538
一、特征衍生 二、特征选择 利用IV值做特征选择 2.1 WOE 2.2 IV 一 特征衍生：利用个人（专家）经验来提取出数据里对结果影响更大的特征，往往是原有数据特征字段通过加减乘除等操作生成新的字段，这些字段在结合一些线性算法做训练的时候往往能起到提升模型效果的作用。举例，用户点外卖，同一家店A，统计一个星期内用户在这家店的购买次数，甚至统计一个月时间范围内，这样的时间周期特征就可以被...
算法实战应用案例精讲-特征工程| 图解机器学习
qq_36130719的博客
09-07 402
首先我们来了解一下『特征工程』。事实上大家在ShowMeAI的实战系列文章 Python机器学习综合项目-电商销量预估[2] 和 Python机器学习综合项目-电商销量预估(进阶)[3] 中已经看到了我们做了特征工程的处理。如果我们对特征工程（feature engineering）做一个定义，那它指的是：利用领域知识和现有数据，创造出新的特征，用于机器学习算法；可以手动或自动。在业界有一个很流行的说法：据与特征工程决定了模型的上限，改进算法只不过是逼近这个上限而已。
Python特征工程：时间特征构造以及实战
weixin_43373042的博客
02-08 449
来源丨数据STUDIO数据和特征决定了机器学习的上限，而模型和算法只是逼近这个上限而已。由此可见，特征工程在机器学习中占有相当重要的地位。在实际应用当中，可以说特征工程是机器学习成功的关键。那特征工程是什么？特征工程是利用数据领域的相关知识来创建能够使机器学习算法达到最佳性能的特征的过程。特征工程又包含了 Data PreProcessing（数据预处理）、Feature Extraction（特...
Python特征工程
weixin_45366750的博客
08-12 2602
一、目的 将数据转换为能更好地表示潜在问题的特征，从而提高机器学习的性能 二、内容 缺失：加入先验知识 异常：清除噪声 集中：数据集中（归一化、标准化）、变量集中（特征筛选、降维） 有价值：数据有价值（数据分桶）、变量有价值（特征构造） ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值