机器学习实战——特征工程(下）

最新推荐文章于 2022-12-05 23:11:40 发布

Code进阶狼人

最新推荐文章于 2022-12-05 23:11:40 发布

阅读量373

点赞数 1

分类专栏： Python机器学习与深度学习文章标签：机器学习

本文链接：https://blog.csdn.net/liuzuoping/article/details/103335670

版权

Python机器学习与深度学习专栏收录该内容

37 篇文章 18 订阅

订阅专栏

上篇博文我们一起学习了特征工程
机器学习实战——特征工程(上）
现在我们对招聘数据进行特征工程探索
具体代码与源文件可以从我的GitHub地址获取
https://github.com/liuzuoping/MeachineLearning-Case
欢迎star~

7.招聘数据的特征工程探索

import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd

读取源数据

lagou_df = pd.read_csv('./lagou_data5.csv', encoding='gbk')
lagou_df.head()

在这里插入图片描述

# advantage和label这两个特征作用不大，可在最后剔除
# 分类变量one-hot处理
# pandas one-hot方法
pd.get_dummies(lagou_df['city']).head()

在这里插入图片描述

# sklearn onehot方法
# 先要硬编码labelcoder
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import LabelEncoder
lbl = LabelEncoder()
lbl.fit(list(lagou_df['city'].values))
lagou_df['city'] = lbl.transform(list(lagou_df['city'].values))
# 查看硬编码结果
lagou_df['city'].head()

在这里插入图片描述

# 再由硬编码转为one-hot编码
df_city = OneHotEncoder().fit_transform(lagou_df['city'].values.reshape((-1,1))).toarray()
df_city[:5]

在这里插入图片描述

# 分类特征统一one-hot处理
cat_features = ['city', 'industry', 'education', 'position_name', 'size', 'stage', 'work_year']
for col in cat_features:
    temp = pd.get_dummies(lagou_df[col])
    lagou_df = pd.concat([lagou_df, temp],axis=1)
    lagou_df = lagou_df.drop([col], axis=1)
    
lagou_df.shape

(1650, 54)

pd.options.display.max_columns = 999
lagou_df = lagou_df.drop(['advantage', 'label'], axis=1)
lagou_df.head()

在这里插入图片描述

职位描述特征的信息提取

lagou_df2 = pd.read_csv('./lagou_data5.csv', encoding='gbk')
lagou_df2 = lagou_df2[['position_detail', 'salary']]
# 提取Python信息
for i, j in enumerate(lagou_df2['position_detail']):
    if 'python' in j:
        lagou_df2['position_detail'][i] = j.replace('python', 'Python')
lagou_df2['Python'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'Python' in j:
        lagou_df2['Python'][i] = 1
    else:
        lagou_df2['Python'][i] = 0
        
lagou_df2['Python'][:20]

lagou_df2['R'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'R' in j:
        lagou_df2['R'][i] = 1
    else:
        lagou_df2['R'][i] = 0
        
lagou_df2['R'].value_counts()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'sql' in j:
        lagou_df2['position_detail'][i] = j.replace('sql', 'SQL')

lagou_df2['SQL'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'SQL' in j:
        lagou_df2['SQL'][i] = 1
    else:
        lagou_df2['SQL'][i] = 0
        
lagou_df2['SQL'].value_counts()
lagou_df2['Excel'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'Excel' in j:
        lagou_df2['Excel'][i] = 1
    else:
        lagou_df2['Excel'][i] = 0
        
lagou_df2['Excel'].value_counts()
lagou_df2['Java'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'Java' in j:
        lagou_df2['Java'][i] = 1
    else:
        lagou_df2['Java'][i] = 0
        
lagou_df2['Java'].value_counts()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'linux' in j:
        lagou_df2['position_detail'][i] = j.replace('linux', 'Linux')
        
lagou_df2['Linux'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'Linux' in j:
        lagou_df2['Linux'][i] = 1
    else:
        lagou_df2['Linux'][i] = 0
        
lagou_df2['Linux'].value_counts()
lagou_df2['C++'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'C++' in j:
        lagou_df2['C++'][i] = 1
    else:
        lagou_df2['C++'][i] = 0
        
lagou_df2['C++'].value_counts()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'spark' in j:
        lagou_df2['position_detail'][i] = j.replace('spark', 'Spark')

lagou_df2['Spark'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'Spark' in j:
        lagou_df2['Spark'][i] = 1
    else:
        lagou_df2['Spark'][i] = 0
        
lagou_df2['Spark'].value_counts()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'tensorflow' in j:
        lagou_df2['position_detail'][i] = j.replace('tensorflow', 'Tensorflow')
        
    if 'TensorFlow' in j:
        lagou_df2['position_detail'][i] = j.replace('TensorFlow', 'Tensorflow')
        
lagou_df2['Tensorflow'] = pd.Series()
for i, j in enumerate(lagou_df2['position_detail']):
    if 'Tensorflow' in j:
        lagou_df2['Tensorflow'][i] = 1
    else:
        lagou_df2['Tensorflow'][i] = 0
        
lagou_df2['Tensorflow'].value_counts()
lagou_df2 = lagou_df2.drop(['position_detail'], axis=1)
lagou_df2.head()