『R语言&Python』针对多分类离散型变量的特征工程

最新推荐文章于 2024-07-17 21:04:45 发布

Python捞数人

最新推荐文章于 2024-07-17 21:04:45 发布

阅读量855

点赞数

分类专栏： R语言 Python 文章标签： python

本文链接：https://blog.csdn.net/m0_47149835/article/details/120481830

版权

Python 同时被 2 个专栏收录

21 篇文章 2 订阅

订阅专栏

R语言

5 篇文章 0 订阅

订阅专栏

在训练模型之前，我们常常需要根据不同变量的基本情况进行相应且合理的特征工程，通过阅读文献和自行尝试，我针对多分类变量的特征工程做出了一些总结
$数据来源（ a d u l t 数据集）： h t t p s : / / a r c h i v e . i c s . u c i . e d u / m l / d a t a s e t s / A d u l t$
也可以直接下载我整理过来用
链接：https://pan.baidu.com/s/1UhGTfvZqPHUC6jnukfTcRg
提取码：j4C9

$P y t h o n$

首先来看看下数据集的基本情况

import pandas as pd
import numpy as np
file = 'C:/Varian/Data_of_training_model/adult/train.csv'
data = pd.read_csv(file, sep=',')

# 首先用上一篇文章中写的函数获取下连续型和离散型变量
def classify(dataframe):
    continuous_variables = []
    categorical_variables = []
    for i in dataframe.columns:
        if data[i] .dtypes == object:
            categorical_variables.append(i)
        else:
            continuous_variables.append(i)
    return continuous_variables, categorical_variables
            
continuous_variables, categorical_variables = classify(data)
print(continuous_variables)
'''
['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', 'hours-per-week']
'''
print(categorical_variables)
'''
['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country', 'income']
'''

可以看出这个数据集中包含了非常多的离散型变量

# 再看看每个分类变量包含了多少类别
for i in categorical_variables:
    print('variable_name: {} \n {} \n category_number: {} \n'.format(i, data[i].value_counts(), len(data[i].value_counts())))

 variable_name:  workclass 
  Private             33906
 Self-emp-not-inc     3862
 Local-gov            3136
 ?                    2799
 State-gov            1981
 Self-emp-inc         1695
 Federal-gov          1432
 Without-pay            21
 Never-worked           10
Name: workclass, dtype: int64 
 category_number:  9 

variable_name:  education 
  HS-grad         15784
 Some-college    10878
 Bachelors        8025
 Masters          2657
 Assoc-voc        2061
 11th             1812
 Assoc-acdm       1601
 10th             1389
 7th-8th           955
 Prof-school       834
 9th               756
 12th              657
 Doctorate         594
 5th-6th           509
 1st-4th           247
 Preschool          83
Name: education, dtype: int64 
 category_number:  16 

variable_name:  marital-status 
  Married-civ-spouse       22379
 Never-married            16117
 Divorced                  6633
 Separated                 1530
 Widowed                   1518
 Married-spouse-absent      628
 Married-AF-spouse           37
Name: marital-status, dtype: int64 
 category_number:  7 

variable_name:  occupation 
  Prof-specialty       6172
 Craft-repair         6112
 Exec-managerial      6086
 Adm-clerical         5611
 Sales                5504
 Other-service        4923
 Machine-op-inspct    3022
 ?                    2809
 Transport-moving     2355
 Handlers-cleaners    2072
 Farming-fishing      1490
 Tech-support         1446
 Protective-serv       983
 Priv-house-serv       242
 Armed-Forces           15
Name: occupation, dtype: int64 
 category_number:  15 

variable_name:  relationship 
  Husband           19716
 Not-in-family     12583
 Own-child          7581
 Unmarried          5125
 Wife               2331
 Other-relative     1506
Name: relationship, dtype: int64 
 category_number:  6 

variable_name:  race 
  White                 41762
 Black                  4685
 Asian-Pac-Islander     1519
 Amer-Indian-Eskimo      470
 Other                   406
Name: race, dtype: int64 
 category_number:  5 

variable_name:  sex 
  Male      32650
 Female    16192
Name: sex, dtype: int64 
 category_number:  2 

variable_name:  native-country 
  United-States                 43832
 Mexico                          951
 ?                               857
 Philippines                     295
 Germany                         206
 Puerto-Rico                     184
 Canada                          182
 El-Salvador                     155
 India                           151
 Cuba                            138
 England                         127
 China                           122
 South                           115
 Jamaica                         106
 Italy                           105
 Dominican-Republic              103
 Japan                            92
 Guatemala                        88
 Poland                           87
 Vietnam                          86
 Columbia                         85
 Haiti                            75
 Portugal                         67
 Taiwan                           65
 Iran                             59
 Nicaragua                        49
 Greece                           49
 Peru                             46
 Ecuador                          45
 France                           38
 Ireland                          37
 Hong                             30
 Thailand                         30
 Cambodia                         28
 Trinadad&Tobago                  27
 Outlying-US(Guam-USVI-etc)       23
 Yugoslavia                       23
 Laos                             23
 Scotland                         21
 Honduras                         20
 Hungary                          19
 Holand-Netherlands                1
Name: native-country, dtype: int64 
 category_number:  42 

variable_name:  income 
  <=50K     24720
 <=50K.    12435
 >50K       7841
 >50K.      3846
Name: income, dtype: int64 
 category_number:  4

一看结果，好家伙，大部分离散型变量的类别数都很多。而且因变量 $i n c o m e$ 由于我合并了训练集和验证集，包含了四个类别，正常应为两个，因此可以使用DataFrame.replace(to_replace=None, value=None, inplace=False, limit=None, regex=False, method='pad')处理掉

修改某一列的值

data['income'].replace(' >50K.', ' >50K', inplace = True)
data['income'].replace(' <=50K.', ' <=50K', inplace = True)
print(data['income'].value_counts())
'''
 <=50K    37155
 >50K     11687
Name: income, dtype: int64
'''

使用字典合并类别

变量 $e d u c a t i o n$ 也包含了很多类别，但实际上可以把部分类别归为一类，使用DataFrame.map(arg, na_action=None)函数再配合字典就能实现这一功能

原始类别 -> 新类别
Preschool -> Dropout
10th -> Dropout
11th -> Dropout
12th -> Dropout
1st-4th -> Dropout
5th-6th -> Dropout
7th-8th -> Dropout
9th -> Dropout
HS-Grad -> HighGrad
Some-college -> Community
Assoc-acdm -> Community
Assoc-voc -> Community
Bachelors -> Bachelors
Masters -> Masters
Prof-school -> Masters
Doctorate -> PhD

data['education'] = data['education'].map({' Preschool':' Dropout',
											' 10th' : ' Dropout',
											' 11th' : ' Dropout',
											' 12th' : ' Dropout',
											' 1st-4th' : ' Dropout',
											' 5th-6th' : ' Dropout',
											' 7th-8th' : ' Dropout',
											' 9th'  : ' Dropout',
											' HS-goad' : ' HighGrad',   # 这里故意把 HS-grad写错为 HS-goad，为了生成NaN，方便后面做演示
											' Some-college' : ' Community',
											' Assoc-acdm' : ' Community',
											' Assoc-voc' : ' Community',
											' Bachelors' : ' Bachelors',
											' Masters' : ' Masters',
											' Prof-school' : ' Masters',
											' Doctorate' : ' PhD'})
print(data['education'].value_counts())
'''
 Community    14540
 Bachelors     8025
 Dropout       6408
 Masters       3491
 PhD            594
Name: education, dtype: int64
'''

转为虚拟变量

转为虚拟变量时需要注意的是：是否需要考虑多重共线性，若你想建立的是回归模型，那么答案是肯定的，这时你除了转换还需要对所生成的虚拟变量做删列处理（加参数drop_first = True）；若不是，那转换即可，不用进行额外操作

# 若建立回归模型
data['education'] = pd.get_dummies(data['education'], prefix = 'education', drop_first = True)
'''
# 对比能发现类别 Bachelors相应的列被删去了
   education_ Community  education_ Dropout  education_ Masters  \
0                     0                   0                   0   
1                     0                   0                   0   
2                     0                   0                   0   
3                     0                   1                   0   
4                     0                   0                   0   
5                     0                   0                   1   
6                     0                   1                   0   
7                     0                   0                   0   
8                     0                   0                   1   
9                     0                   0                   0   

   education_ PhD  
0               0  
1               0  
2               0  
3               0  
4               0  
5               0  
6               0  
7               0  
8               0  
9               0 
'''

# 用于建立其它模型
data['education'] = pd.get_dummies(data['education'], prefix = 'education')
'''
   education_ Bachelors  education_ Community  education_ Dropout  \
0                     1                     0                   0   
1                     1                     0                   0   
2                     0                     0                   0   
3                     0                     0                   1   
4                     1                     0                   0   
5                     0                     0                   0   
6                     0                     0                   1   
7                     0                     0                   0   
8                     0                     0                   0   
9                     1                     0                   0   

   education_ Masters  education_ PhD  
0                   0               0  
1                   0               0  
2                   0               0  
3                   0               0  
4                   0               0  
5                   1               0  
6                   0               0  
7                   0               0  
8                   1               0  
9                   0               0
'''

根据多列值，使用逻辑关系生成新的列

这里的指的是那种需要多层判断，通常情况下需要可递归函数才能实现的复杂逻辑；简单逻辑(即一层 $i f 、 e l s e$ 就能完成的逻辑)可以使用np.where或者np.select来实现；当然复杂的逻辑也用np.select实现，但它要求你完整地列出所有情况，这会使得代码变得极其冗长且不美观；至于np.where，它支持递归，能实现多层简单逻辑，但是不支持存在 $o r (∣)$ 和 $a n d$ (&)的表达式 (这个后面会给出案例)

那么怎么处理需要多层判断的复杂逻辑呢？当时由于时间紧迫没时间查资料（~~抄代码~~），且被要求尽量不写新函数来实现，所以我只能从本质入手：不过无论逻辑多复杂，只要是生成新的列，都是按照行的维度，在新列中逐个生成新元素

# 选出三列便于演示
selected_col = ['age', 'race', 'education']
test_data = data[selected_col]
# 只看前十行
test_data = test_data.head(10)
test_data
'''
   age    race   education
0   39   White   Bachelors
1   50   White   Bachelors
2   38   White         NaN
3   53   Black     Dropout
4   28   Black   Bachelors
5   37   White     Masters
6   49   Black     Dropout
7   52   White         NaN
8   31   White     Masters
9   42   White   Bachelors
'''

'''
使用以下逻辑生成新的列test_col:
1.当age>37时,若race = white,若education = NaN,则test_col = Out
							否则test_col = Good;
             若race = black,若education = Dropout 或 NaN,则test_col = Out;
                            否则test_col = Good;
2.当age<=37时,若education = masters,则test_col = Best
             若education = bachelors,则test_col = Impressive.
             
# 逻辑随便写的，本人无种族歧视，这里仅作演示
'''
# 首先创建一个空列表来存放 test_col 中的元素
test_col = []
# 根据多列的值和逻辑生成新的列，本质实际是逐行进行逻辑判断
for i in range(len(test_data)):
    if test_data.iloc[i]['age'] > 37:  # 若大于37岁
        if test_data.iloc[i]['race'] == ' White':  # 且为白人
	        if pd.isnull(test_data.iloc[i]['education']): # 若教育为空值
	                test_col.append('Out')
	            else:              	   # 否则其他所有情况都记为Good
	                test_col.append('Good') # 新列记为Good   
        else:                          # 否则，即若为黑人
            if pd.isnull(test_data.iloc[i]['education']) or test_data.iloc[i]['education'] == ' Dropout': # 若没受过教育或值为Dropout
                test_col.append('Out') # 新列记为Out
            else:					   # 除了空值和Dropout的所有值
                test_col.append('Good')# 新列记为Good
    else:							   # 否则，即年龄≤37岁
        if test_data.iloc[i]['education'] == ' Masters':    # 若教育值为Masters
             test_col.append('Best')   # 新列记为Best
        else:						   # 除了Masters的所有值
            test_col.append('Impressive') # 新列记为Impressive
# 将列表转为DataFrame            
test_col = pd.DataFrame(test_col, columns=['test_col'])
# 合并DataFrame
test_data = pd.concat([test_data, test_col], axis = 1)
print(test_data)
'''
   age    race   education    test_col
0   39   White   Bachelors        Good
1   50   White   Bachelors        Good
2   38   White         NaN        Good
3   53   Black     Dropout         Out
4   28   Black   Bachelors  Impressive
5   37   White     Masters        Best
6   49   Black     Dropout         Out
7   52   White         NaN        Good
8   31   White     Masters        Best
9   42   White   Bachelors        Good
'''

总结一下：

计算 $D a t a F r a m e$ 长度，遍历赋值给 $i$
使用DataFrame.iloc[i]['colname']代表第 $i$ 行第 $c o l n a m e$ 列元素
利用多层 $i f, e l s e$ 进行逻辑判断

🔺:注意逻辑产生的列的长度必须和原数据长度相等，否则报错

现在让我们看下用np.where(condition, yes, no)怎么实现

selected_col = ['age', 'race', 'education']
test_data = data[selected_col]
test_data = test_data.head(10)
test_data['test_col'] = np.where(test_data['age']>37, 
                                 np.where(test_data['race']==' White', 
                                          np.where(pd.isnull(test_data['education']), 'Out', 'Good'), 
                                          np.where(test_data['education']==' Dropout', 'Out', 'Good')), # 注意这里我没加 or判断是否为空，但结果意外地和上面相同的
                                 np.where(test_data['education']==' Masters', 'Best', 
                                          np.where(test_data['education']==' Bachelors', 'Impressive', 'else')))
# 简洁，但是代码可读性较差
print(test_data)
'''
   age    race   education    test_col
0   39   White   Bachelors        Good
1   50   White   Bachelors        Good
2   38   White         NaN         Out
3   53   Black     Dropout         Out
4   28   Black   Bachelors  Impressive
5   37   White     Masters        Best
6   49   Black     Dropout         Out
7   52   White         NaN         Out
8   31   White     Masters        Best
9   42   White   Bachelors        Good
'''

然后试试它支不支持 $o r (∣)$ 和 $a n d$ (&)表达式：

selected_col = ['age', 'race', 'education']
test_data = data[selected_col]
test_data = test_data.head(10)
print(test_data)
'''
   age    race   education
0   39   White   Bachelors
1   50   White   Bachelors
2   38   White         NaN
3   53   Black     Dropout
4   28   Black   Bachelors
5   37   White     Masters
6   49   Black     Dropout
7   52   White         NaN
8   31   White     Masters
9   42   White   Bachelors
'''
test_data['new_col'] = np.where(test_data['race']==' White'| pd.isnull(test_data['education']), 'Yes', 'No')
# 报错信息
TypeError: ufunc 'bitwise_or' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
----------
test_data['new_col'] = np.where(test_data['race']==' White'& pd.isnull(test_data['education']), 'Yes', 'No')
# 报错信息
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

所以结论是：np.where确实不支持包含 $o r$ 或 $a n d$ 的表达式

而用按行遍历的方法却能轻松实现 $o r$ 或 $a n d$ ：

for i in range(len(test_data)):
    if test_data.iloc[i]['race'] == ' White' or pd.isnull(test_data.iloc[i]['education']):
        new_col.append('Yes')
    else:
        new_col.append('No')
new_col = pd.DataFrame(new_col, columns=['new_col'])
test_data = pd.concat([test_data, new_col], axis=1)
print(test_data)
'''
   age    race   education new_col
0   39   White   Bachelors     Yes
1   50   White   Bachelors     Yes
2   38   White         NaN     Yes
3   53   Black     Dropout      No
4   28   Black   Bachelors      No
5   37   White     Masters     Yes
6   49   Black     Dropout      No
7   52   White         NaN     Yes
8   31   White     Masters     Yes
9   42   White   Bachelors     Yes
'''

Ps:如果 $P y t h o n$ 中有和 $R$ 里 $i f e l s e$ 完全一样功能的函数就好了 (╯▔皿▔)╯

$R$

使用 $R$ 语言中的dplyr包实现上述功能会简单很多

select_if(data, function)能筛选出符合 $f u n c t i o n$ 的列

library(dplyr)
data = read.csv('C:/Varian/Data_of_training_model/adult/train.csv', sep = ',', header = TRUE)
# 查看连续型变量
continuous = select_if(data, is.numeric)
colnames(continuous)
[1] "age"            "fnlwgt"         "education.num"  "capital.gain"  
[5] "capital.loss"   "hours.per.week"
# 查看离散型变量
categorical = select_if(data, is.factor)
colnames(categorical)
[1] "workclass"      "education"      "marital.status" "occupation"    
[5] "relationship"   "race"           "sex"            "native.country"
[9] "income"

修改某一列的值

方式1：

使用mutate()函数修改某列的值，通过case_when()函数来对列中具体的不同分类值做处理

data = data %>%
  mutate(income = case_when(
    income == ' >50K.'|income == ' >50K' ~ '>50k',
    income == ' <=50K.'|income == ' <=50K' ~ '<=50k'
  ))
 table(data$income)
<=50k  >50k 
37155 11687

方式2：

使用ifelse(condition, yes, no)函数（满足 $c o n d i t i o n$ 时，执行或返回 $y e s$ ,否则执行或返回 $n o$ ）

data = read.csv('C:/Varian/Data_of_training_model/adult/train.csv', sep = ',', header = TRUE)
data$income = ifelse(data$income==' >50K.'|data$income==' >50K', '>50k',
                     ifelse(data$income==' <=50K.'|data$income==' <=50K', '<=50k', 'eles'))
table(data$income)
<=50k  >50k 
37155 11687

合并多类为一类

原始类别 -> 新类别
Preschool -> Dropout
10th -> Dropout
11th -> Dropout
12th -> Dropout
1st-4th -> Dropout
5th-6th -> Dropout
7th-8th -> Dropout
9th -> Dropout
HS-Grad -> HighGrad
Some-college -> Community
Assoc-acdm -> Community
Assoc-voc -> Community
Bachelors -> Bachelors
Masters -> Masters
Prof-school -> Masters
Doctorate -> PhD

方式1：

data = data%>%
  mutate(education = case_when(
    education == ' Masters'|education == ' Prof-school' ~ 'Master',
    education == ' Bachelors' ~ 'Bachelors',
    education == ' Assoc-voc'|education == ' Assoc-acdm' |education == ' Some-college' ~ 'Community',
    education == ' HS-grad' ~ 'HighGrad', 
    education == ' Preschool'|education == ' 10th'|education ==' 11th'|education == ' 12th'|education == ' 1st-4th'|education ==' 5th-6th'|education == ' 7th-8th'|education == ' 9th' ~ 'dropout',
    education == ' Doctorate' ~ 'PHD'
      ))
table(data$education)
Bachelors Community   dropout  HighGrad    Master       PHD 
     8025     14540      6408     15784      3491       594

方式2：

使用ifelse(condition, yes, no)函数（满足 $c o n d i t i o n$ 时，执行或返回 $y e s$ ,否则执行或返回 $n o$ ）

data <- data %>%
  mutate(education = factor(ifelse(education == " Preschool" | education == " 10th" | education == " 11th" | education == " 12th" | education == " 1st-4th" | education == " 5th-6th" | education == " 7th-8th" | education == " 9th", " dropout", ifelse(education == " HS-grad", " HighGrad", ifelse(education == " Some-college" | education == " Assoc-acdm" | education == " Assoc-voc", "Community",
                                                                                                                                                                                                                                                                                            ifelse(education == " Bachelors", "Bachelors",
                                                                                                                                                                                                                                                                                                   ifelse(education == " Masters" | education == " Prof-school", "Master", "PhD")))))))
table(data$education)
 dropout  HighGrad Bachelors Community    Master       PhD 
    6408     15784      8025     14540      3491       594

根据多列值，使用逻辑关系生成新的列

还是可以使用ifelse(condition, yes, no)函数

$i f e l s e$ 我吹爆好吧！！！ $n p . w h e r e$ 是神马辣鸡！(╯▔皿▔)╯

data = read.csv('C:/Varian/Data_of_training_model/adult/train.csv', sep = ',', header = TRUE)
data = data%>%
  mutate(education = case_when(
    education == ' Masters'|education == ' Prof-school' ~ ' Masters',
    education == ' Bachelors' ~ ' Bachelors',
    education == ' Assoc-voc'|education == ' Assoc-acdm' |education == ' Some-college' ~ 'Community',
    education == ' HS-goad' ~ 'HighGrad',               # 这里也和python部分一样，故意写错来生成Na，方便后面演示
    education == ' Preschool'|education == ' 10th'|education ==' 11th'|education == ' 12th'|education == ' 1st-4th'|education ==' 5th-6th'|education == ' 7th-8th'|education == ' 9th' ~ ' Dropout',
    education == ' Doctorate' ~ 'PHD'
  ))
test_data = subset(data, select = c('age', 'race', 'education'))
test_data = test_data[1:10,]
test_data
test_data$new_col = ifelse(test_data$age>37, 
                           ifelse(test_data$race==' White',
                                  ifelse(is.na(test_data$education), 'Out', 'Good'),
                                  ifelse(test_data$education==' Dropout'|is.na(test_data$education), 'Out', 'Good')),
                           ifelse(test_data$education==' Masters', 'Best', 'Impressive'))
print(test_data)
    X age   race  education    new_col
1   1  39  White  Bachelors       Good
2   2  50  White  Bachelors       Good
3   3  38  White       <NA>        Out
4   4  53  Black    Dropout        Out
5   5  28  Black  Bachelors Impressive
6   6  37  White    Masters       Best
7   7  49  Black    Dropout        Out
8   8  52  White       <NA>        Out
9   9  31  White    Masters       Best
10 10  42  White  Bachelors       Good

# 和python部分的结果一样！完美！

转为虚拟变量

在逻辑回归glm(..., family = 'binomial')中，模型会自动帮我们将类型为 $f a c t o r$ 的变量都转为虚拟变量并消除共线性(删除第一列)后才开始计算，其它回归模型我暂时没有去深入研究

当然也可以使用dummies包内的dummy函数生成虚拟变量：

library(dummies)
test = c(1,3,3,1,1,1,2)
dummy(test, sep = '_')
     test_1 test_2 test_3
[1,]      1      0      0
[2,]      0      0      1
[3,]      0      0      1
[4,]      1      0      0
[5,]      1      0      0
[6,]      1      0      0
[7,]      0      1      0

Python捞数人

关注

0
点赞
踩
6

收藏

觉得还不错? 一键收藏
0
评论
『R语言&Python』针对多分类离散型变量的特征工程

在训练模型之前，我们常常需要根据不同变量的基本情况进行相应且合理的特征工程，通过阅读文献和自行尝试，我针对多分类变量的特征工程做出了一些总结数据来源（adult数据集）：https://archive.ics.uci.edu/ml/datasets/Adult数据来源（adult数据集）：https://archive.ics.uci.edu/ml/datasets/Adult数据来源（adult数据集）：https://archive.ics.uci.edu/ml/datasets/Adult P.
复制链接

扫一扫