机器学习：美国50K工资分类（改）

最新推荐文章于 2023-05-15 23:49:51 发布
最新推荐文章于 2023-05-15 23:49:51 发布
阅读量2.4k 收藏 4
点赞数
分类专栏： python数据分析总结 文章标签： python 机器学习
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_33489955/article/details/124210791
版权

目录


之前写过一次，这篇是再修改版本（ 前版本）

1.观察并导入数据

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

plt.rcParams['font.sans-serif']=['SimHei'] #用来正常显示中文标签
plt.rcParams['axes.unicode_minus'] = False #用来正常显示负号

df = pd.read_csv('adults.csv')
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 32561 entries, 0 to 32560
Data columns (total 15 columns):
 #   Column          Non-Null Count  Dtype 
---  ------          --------------  ----- 
 0   age             32561 non-null  int64 
 1   workclass       32561 non-null  object
 2   final_weight    32561 non-null  int64 
 3   education       32561 non-null  object
 4   education_num   32561 non-null  int64 
 5   marital_status  32561 non-null  object
 6   occupation      32561 non-null  object
 7   relationship    32561 non-null  object
 8   race            32561 non-null  object
 9   sex             32561 non-null  object
 10  capital_gain    32561 non-null  int64 
 11  capital_loss    32561 non-null  int64 
 12  hours_per_week  32561 non-null  int64 
 13  native_country  32561 non-null  object
 14  salary          32561 non-null  object
dtypes: int64(6), object(9)
memory usage: 3.7+ MB

df.head()
ageworkclassfinal_weighteducationeducation_nummarital_statusoccupationrelationshipracesexcapital_gaincapital_losshours_per_weeknative_countrysalary
039State-gov77516Bachelors13Never-marriedAdm-clericalNot-in-familyWhiteMale2174040United-States<=50K
150Self-emp-not-inc83311Bachelors13Married-civ-spouseExec-managerialHusbandWhiteMale0013United-States<=50K
238Private215646HS-grad9DivorcedHandlers-cleanersNot-in-familyWhiteMale0040United-States<=50K
353Private23472111th7Married-civ-spouseHandlers-cleanersHusbandBlackMale0040United-States<=50K
428Private338409Bachelors13Married-civ-spouseProf-specialtyWifeBlackFemale0040Cuba<=50K
df.workclass.value_counts()

Private             22696
Self-emp-not-inc     2541
Local-gov            2093
?                    1836
State-gov            1298
Self-emp-inc         1116
Federal-gov           960
Without-pay            14
Never-worked            7
Name: workclass, dtype: int64

df.capital_gain.value_counts()

0        29849
15024      347
7688       284
7298       246
99999      159
         ...  
1111         1
2538         1
22040        1
4931         1
5060         1
Name: capital_gain, Length: 119, dtype: int64

df.salary.value_counts()

<=50K    24720
>50K      7841
Name: salary, dtype: int64

# 过采样 重复少的/收集新的/算法伪造
# 欠采样 抛弃多的数据
#

2. 处理数据

缺失值处理

# 缺失值都是用  ? 替换的
df = df.replace('?',np.nan)
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 32561 entries, 0 to 32560
Data columns (total 15 columns):
 #   Column          Non-Null Count  Dtype 
---  ------          --------------  ----- 
 0   age             32561 non-null  int64 
 1   workclass       30725 non-null  object
 2   final_weight    32561 non-null  int64 
 3   education       32561 non-null  object
 4   education_num   32561 non-null  int64 
 5   marital_status  32561 non-null  object
 6   occupation      30718 non-null  object
 7   relationship    32561 non-null  object
 8   race            32561 non-null  object
 9   sex             32561 non-null  object
 10  capital_gain    32561 non-null  int64 
 11  capital_loss    32561 non-null  int64 
 12  hours_per_week  32561 non-null  int64 
 13  native_country  31978 non-null  object
 14  salary          32561 non-null  object
dtypes: int64(6), object(9)
memory usage: 3.7+ MB

2.1探索数据（对df进行数据探索，df为将’?’ 替换为np.nan 的数据集，clean_df 为删除了’?'所在行的数据集）

import seaborn as sns
%matplotlib inline

#检查数据相关性
sns.heatmap(df.corr()
           ,annot=True
           ,center=0
           ,linewidth=0.8)

<AxesSubplot:>

在这里插入图片描述

直接检查相关性似乎并不妥当，因为数据并不都是数值类型

查看数值类型的特征：

df.describe([0.01,0.99]).T
countmeanstdmin1%50%99%max
age32561.038.58164713.64043317.017.037.074.090.0
final_weight32561.0189778.366512105549.97769712285.027185.8178356.0510072.01484705.0
education_num32561.010.0806792.5727201.03.010.016.016.0
capital_gain32561.01077.6488447385.2920850.00.00.015024.099999.0
capital_loss32561.087.303830402.9602190.00.00.01980.04356.0
hours_per_week32561.040.43745612.3474291.08.040.080.099.0
df.describe([0.01,0.99]).T.index

Index(['age', 'final_weight', 'education_num', 'capital_gain', 'capital_loss',
       'hours_per_week'],
      dtype='object')

# 年龄分布
sns.set_style('whitegrid')
plt.subplots(figsize=(15,8))
s = df['age'].value_counts()
sns.barplot(s.index,s.values)

<AxesSubplot:>

在这里插入图片描述

df.age.mean() # 平均年龄

38.58164675532078



# 第一，年龄和工资的关系
s=df['age'].value_counts()
k=df['age'][df['salary']=='>50K'].value_counts()
sns.set_style("whitegrid")
f, ax = plt.subplots(figsize=(18, 9))
sns.set_color_codes("pastel")
sns.barplot(s.index,s.values,label='total',color="b")
sns.barplot(k.index,k.values,label='income>50K',color="g")
ax.legend(ncol=2, loc="upper left", frameon=True)

<matplotlib.legend.Legend at 0x16cd08a8be0>

在这里插入图片描述

# 第二，教育水平
# 高中毕业人数有将近1.6万人，其次是大学肄业
plt.subplots(figsize=(15,6))
s = df['education'].value_counts()
sns.barplot(s.index,s.values)

<AxesSubplot:>

在这里插入图片描述

df['education_num'].value_counts()

9     10501
10     7291
13     5355
14     1723
11     1382
7      1175
12     1067
6       933
4       646
15      576
5       514
8       433
16      413
3       333
2       168
1        51
Name: education_num, dtype: int64

edu_n = df.groupby(['education','education_num'])['education'].count()
edu_n

education     education_num
10th          6                  933
11th          7                 1175
12th          8                  433
1st-4th       2                  168
5th-6th       3                  333
7th-8th       4                  646
9th           5                  514
Assoc-acdm    12                1067
Assoc-voc     11                1382
Bachelors     13                5355
Doctorate     16                 413
HS-grad       9                10501
Masters       14                1723
Preschool     1                   51
Prof-school   15                 576
Some-college  10                7291
Name: education, dtype: int64

education 和 educational-num 是一一对应关系，做算法模型时可以删除一列。

education是分类变量，educational-num是数值变量，受教育水平是有顺序的，数字大小是有比较意义的，因此保留education-num。

s

HS-grad         10501
Some-college     7291
Bachelors        5355
Masters          1723
Assoc-voc        1382
11th             1175
Assoc-acdm       1067
10th              933
7th-8th           646
Prof-school       576
9th               514
12th              433
Doctorate         413
5th-6th           333
1st-4th           168
Preschool          51
Name: education, dtype: int64

edu_hsalary = df['education'][df['salary']=='>50K'].value_counts()
edu_lsalary = df['education'][df['salary']=='<=50K'].value_counts()

edu_high_percent = edu_hsalary/s
edu_low_percent = edu_lsalary/s

fig = plt.figure(figsize=(15,6))
sns.barplot(edu_high_percent.index,edu_high_percent.values,color='red',label='edu_high_salary')
sns.barplot(edu_low_percent.index,edu_low_percent.values,bottom = edu_high_percent,color='yellow',label='edu_low_salary')

fig.legend(ncol=2,loc='upper center',frameon=True)

<matplotlib.legend.Legend at 0x16cd4147d30>

在这里插入图片描述

可以看到超过50K占比最高依次是,Prof-school，Doctorate,Masters,Bachelors 跟学习年限呈现正相关性。

# 第三，婚姻状况
f,ax = plt.subplots(figsize=(11,6))
s=df['marital_status'].value_counts()
sns.barplot(s.index,s.values)
ax.set_xticklabels(ax.get_xticklabels(),rotation=45)

[Text(0, 0, 'Married-civ-spouse'),
 Text(1, 0, 'Never-married'),
 Text(2, 0, 'Divorced'),
 Text(3, 0, 'Separated'),
 Text(4, 0, 'Widowed'),
 Text(5, 0, 'Married-spouse-absent'),
 Text(6, 0, 'Married-AF-spouse')]

在这里插入图片描述

mer_hsalary=df['marital_status'][df['salary']=='>50K'].value_counts()
mer_lsalary=df['marital_status'][df['salary']=='<=50K'].value_counts()

f,ax = plt.subplots(figsize=[12,6])

sns.barplot(mer_hsalary.index,mer_hsalary.values,color='blue',alpha = 0.7,label='mer_low_salary')
sns.barplot(mer_lsalary.index,mer_lsalary.values,color='yellow',alpha = 0.5,label='mer_high_salary')

ax.legend(ncol=2,loc='upper center',frameon=True)

<matplotlib.legend.Legend at 0x16cd41a1b50>

在这里插入图片描述

可以看到已婚和收入高低有正相关性，但还不能说明因此有因果关系。

# 第四、职业分布以及和收入关系
plt.subplots(figsize=(8,8))
s=df['occupation'].value_counts()
sns.barplot(y=s.index,x=s.values)

<AxesSubplot:>

在这里插入图片描述

从业人数前三分别是Prof-specialty,Craft-repair,Exec-managerial

occ_hsalary=df['occupation'][df['salary']=='>50K'].value_counts()
occ_lsalary=df['occupation'][df['salary']=='<=50K'].value_counts()


f,ax=plt.subplots(figsize=[12,6])
a=s.index
sns.barplot(y=occ_hsalary.index,x=occ_hsalary.values,color='yellow',order=a,alpha = 0.7,label='occ_low_salary')

sns.barplot(y=occ_lsalary.index,x=occ_lsalary.values,color='blue',order=a,alpha=0.5,label='occ_high_salary')

ax.legend(ncol=2, loc="lower center", frameon=True)

<matplotlib.legend.Legend at 0x16cd4669100>

在这里插入图片描述

可以看出高收入占比比较高的是Exec-managerial、Prof-specialty，比较低的是Handlers-cleaners、Farming-fishing，比较符合我们的日常认知。

# 5 家庭
plt.subplots(figsize=(8,8))
s=df['relationship'].value_counts()
plt.pie(s.values,labels=s.index,autopct='%1.1f%%')

([<matplotlib.patches.Wedge at 0x16cd4114910>,
  <matplotlib.patches.Wedge at 0x16cd4119580>,
  <matplotlib.patches.Wedge at 0x16cd411f550>,
  <matplotlib.patches.Wedge at 0x16cd4126070>,
  <matplotlib.patches.Wedge at 0x16cd4126fd0>,
  <matplotlib.patches.Wedge at 0x16cd412bac0>],
 [Text(0.3228564400293403, 1.0515530034817937, 'Husband'),
  Text(-1.0768528506930521, -0.22447257728784475, 'Not-in-family'),
  Text(-0.08243982298868217, -1.0969064114980798, 'Own-child'),
  Text(0.7469351484545255, -0.8075195873805269, 'Unmarried'),
  Text(1.0368150199444963, -0.3674433485824635, 'Wife'),
  Text(1.0950764363882008, -0.10395960016909805, 'Other-relative')],
 [Text(0.1761035127432765, 0.5735743655355238, '40.5%'),
  Text(-0.5873742821962102, -0.12243958761155167, '25.5%'),
  Text(-0.04496717617564482, -0.5983125880898617, '15.6%'),
  Text(0.40741917188428656, -0.4404652294802873, '10.6%'),
  Text(0.5655354654242707, -0.20042364468134374, '4.8%'),
  Text(0.5973144198481095, -0.05670523645587165, '3.0%')])

在这里插入图片描述

# 种族
plt.subplots(figsize=(8,8))
s=df['race'].value_counts()
plt.pie(s.values,labels=s.index,autopct='%1.1f%%')

([<matplotlib.patches.Wedge at 0x16cd4114a90>,
  <matplotlib.patches.Wedge at 0x16cd3f28550>,
  <matplotlib.patches.Wedge at 0x16cd3f28dc0>,
  <matplotlib.patches.Wedge at 0x16cd07c7d90>,
  <matplotlib.patches.Wedge at 0x16cd3e227c0>],
 [Text(-0.9867232454903924, 0.486186421868101, 'White'),
  Text(0.898950111132619, -0.6339469202501515, 'Black'),
  Text(1.0752451188481307, -0.23205157701095105, 'Asian-Pac-Islander'),
  Text(1.0962767570479313, -0.0904282696753132, 'Amer-Indian-Eskimo'),
  Text(1.0996240020243266, -0.028758549546243486, 'Other')],
 [Text(-0.5382126793583958, 0.2651925937462369, '85.4%'),
  Text(0.4903364242541558, -0.3457892292273554, '9.6%'),
  Text(0.5864973375535257, -0.12657358746051875, '3.2%'),
  Text(0.5979691402079624, -0.04932451073198901, '1.0%'),
  Text(0.5997949101950872, -0.015686481570678264, '0.8%')])

在这里插入图片描述

# 6 性别
df['sex'].value_counts()

Male      21790
Female    10771
Name: sex, dtype: int64

# 7 资本收益和损失
plt.subplots(figsize=(7,5))
sns.distplot(df['capital_loss'][df['capital_loss']!=0])

<AxesSubplot:xlabel='capital_loss', ylabel='Density'>

在这里插入图片描述

资本损失密度函数，峰值在2000左右，呈现正态分布

plt.subplots(figsize=(7,5))
sns.distplot(df['capital_gain'][df['capital_gain']!=0])

<AxesSubplot:xlabel='capital_gain', ylabel='Density'>

在这里插入图片描述

收益普遍比较高，在10万美金出现很多极值

df['capital_gain'][(df['capital_gain']!=0)|df['capital_loss']].agg(['mean','count'])

mean     8293.387852
count    4231.000000
Name: capital_gain, dtype: float64

df['capital_loss'][(df['capital_gain']!=0)|df['capital_loss']].agg(['mean','count'])

mean      671.874261
count    4231.000000
Name: capital_loss, dtype: float64

在有资本收益损失的调查人群6317人中，其中资本获益人均8343，资本损失人均676，看来行情不错。

gain_age=df.groupby(['age'])['capital_gain'].sum()
loss_age=df.groupby(['age'])['capital_loss'].sum()
fig =plt.figure(figsize=(15,6))
sns.barplot(gain_age.index,gain_age.values,color='yellow',alpha=0.7,label='capital_gain')
sns.barplot(loss_age.index,loss_age.values,color='red',alpha = 0.3,label='capital_loss')
fig.legend(ncol=2, loc="upper center", frameon=True)

<matplotlib.legend.Legend at 0x16cd3f471f0>

在这里插入图片描述

看下分年龄的资本收入情况对比。

# 8 工作时长
plt.subplots(figsize=(9,5))
sns.distplot(df['hours_per_week'])

<AxesSubplot:xlabel='hours_per_week', ylabel='Density'>

在这里插入图片描述

接近50%的人是每周工作40个小时。

plt.subplots(figsize=(8, 9))
s=df['native_country'][df['native_country']!='United-States'].value_counts()
sns.barplot(y=s.index,x=
            s.values)

<AxesSubplot:>

在这里插入图片描述

可以看到前三移民来源国是Mexico，Philippines，Germany

2.2缺失值情况分析

import missingno as msno # 可视化缺失值的库

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 32561 entries, 0 to 32560
Data columns (total 15 columns):
 #   Column          Non-Null Count  Dtype 
---  ------          --------------  ----- 
 0   age             32561 non-null  int64 
 1   workclass       30725 non-null  object
 2   final_weight    32561 non-null  int64 
 3   education       32561 non-null  object
 4   education_num   32561 non-null  int64 
 5   marital_status  32561 non-null  object
 6   occupation      30718 non-null  object
 7   relationship    32561 non-null  object
 8   race            32561 non-null  object
 9   sex             32561 non-null  object
 10  capital_gain    32561 non-null  int64 
 11  capital_loss    32561 non-null  int64 
 12  hours_per_week  32561 non-null  int64 
 13  native_country  31978 non-null  object
 14  salary          32561 non-null  object
dtypes: int64(6), object(9)
memory usage: 3.7+ MB

三列有缺失,workclass,occupation,native_country

msno.bar(df)

<AxesSubplot:>

在这里插入图片描述

绘制缺失值热力图。利用热力图可以观察多个特征两两的相似度，相似度由皮尔逊相关系数度量。

occupation和workclass为1表明这两个变量在数据集中是同步缺失的。

msno.heatmap(df,figsize=(3,2))

<AxesSubplot:>

在这里插入图片描述

分别对训练集和测试集绘制缺失值矩阵图。矩阵图中白线越多，代表缺失值越多。

结果表明workclass和occupation相比于native_country有更多的缺失值

msno.matrix(df,figsize=(6,3))

<AxesSubplot:>

在这里插入图片描述

工作类型workclass和职业occupation分别有5.63%和5.66%的缺失，原籍native_country有1.79%缺失

temp = df.apply(lambda x:x.isna().sum()/len(x))
# temp = temp.loc[:,(temp!=0).any()]
temp

age               0.000000
workclass         0.056386
final_weight      0.000000
education         0.000000
education_num     0.000000
marital_status    0.000000
occupation        0.056601
relationship      0.000000
race              0.000000
sex               0.000000
capital_gain      0.000000
capital_loss      0.000000
hours_per_week    0.000000
native_country    0.017905
salary            0.000000
dtype: float64

temp.plot(kind='bar',figsize=(8,4))

<AxesSubplot:>

在这里插入图片描述

2.3数据处理 及特征工程

# 判断哪些字段不是数值类型
# X = df.iloc[:,:-1].copy()
# y = df.iloc[:,-1].copy()
# for col_name in X:
#     col_data = X[col_name]
#     if col_data.dtype == 'object':
#         print(col_name,'------->',col_data.unique())

"""
workclass -------> ['State-gov' 'Self-emp-not-inc' 'Private' 'Federal-gov' 'Local-gov' nan
 'Self-emp-inc' 'Without-pay' 'Never-worked']
education -------> ['Bachelors' 'HS-grad' '11th' 'Masters' '9th' 'Some-college' 'Assoc-acdm'
 'Assoc-voc' '7th-8th' 'Doctorate' 'Prof-school' '5th-6th' '10th'
 '1st-4th' 'Preschool' '12th']
marital_status -------> ['Never-married' 'Married-civ-spouse' 'Divorced' 'Married-spouse-absent'
 'Separated' 'Married-AF-spouse' 'Widowed']
occupation -------> ['Adm-clerical' 'Exec-managerial' 'Handlers-cleaners' 'Prof-specialty'
 'Other-service' 'Sales' 'Craft-repair' 'Transport-moving'
 'Farming-fishing' 'Machine-op-inspct' 'Tech-support' nan
 'Protective-serv' 'Armed-Forces' 'Priv-house-serv']
relationship -------> ['Not-in-family' 'Husband' 'Wife' 'Own-child' 'Unmarried' 'Other-relative']
race -------> ['White' 'Black' 'Asian-Pac-Islander' 'Amer-Indian-Eskimo' 'Other']
sex -------> ['Male' 'Female']
native_country -------> ['United-States' 'Cuba' 'Jamaica' 'India' nan 'Mexico' 'South'
 'Puerto-Rico' 'Honduras' 'England' 'Canada' 'Germany' 'Iran'
 'Philippines' 'Italy' 'Poland' 'Columbia' 'Cambodia' 'Thailand' 'Ecuador'
 'Laos' 'Taiwan' 'Haiti' 'Portugal' 'Dominican-Republic' 'El-Salvador'
 'France' 'Guatemala' 'China' 'Japan' 'Yugoslavia' 'Peru'
 'Outlying-US(Guam-USVI-etc)' 'Scotland' 'Trinadad&Tobago' 'Greece'
 'Nicaragua' 'Vietnam' 'Hong' 'Ireland' 'Hungary' 'Holand-Netherlands']
"""

"\nworkclass -------> ['State-gov' 'Self-emp-not-inc' 'Private' 'Federal-gov' 'Local-gov' nan\n 'Self-emp-inc' 'Without-pay' 'Never-worked']\neducation -------> ['Bachelors' 'HS-grad' '11th' 'Masters' '9th' 'Some-college' 'Assoc-acdm'\n 'Assoc-voc' '7th-8th' 'Doctorate' 'Prof-school' '5th-6th' '10th'\n '1st-4th' 'Preschool' '12th']\nmarital_status -------> ['Never-married' 'Married-civ-spouse' 'Divorced' 'Married-spouse-absent'\n 'Separated' 'Married-AF-spouse' 'Widowed']\noccupation -------> ['Adm-clerical' 'Exec-managerial' 'Handlers-cleaners' 'Prof-specialty'\n 'Other-service' 'Sales' 'Craft-repair' 'Transport-moving'\n 'Farming-fishing' 'Machine-op-inspct' 'Tech-support' nan\n 'Protective-serv' 'Armed-Forces' 'Priv-house-serv']\nrelationship -------> ['Not-in-family' 'Husband' 'Wife' 'Own-child' 'Unmarried' 'Other-relative']\nrace -------> ['White' 'Black' 'Asian-Pac-Islander' 'Amer-Indian-Eskimo' 'Other']\nsex -------> ['Male' 'Female']\nnative_country -------> ['United-States' 'Cuba' 'Jamaica' 'India' nan 'Mexico' 'South'\n 'Puerto-Rico' 'Honduras' 'England' 'Canada' 'Germany' 'Iran'\n 'Philippines' 'Italy' 'Poland' 'Columbia' 'Cambodia' 'Thailand' 'Ecuador'\n 'Laos' 'Taiwan' 'Haiti' 'Portugal' 'Dominican-Republic' 'El-Salvador'\n 'France' 'Guatemala' 'China' 'Japan' 'Yugoslavia' 'Peru'\n 'Outlying-US(Guam-USVI-etc)' 'Scotland' 'Trinadad&Tobago' 'Greece'\n 'Nicaragua' 'Vietnam' 'Hong' 'Ireland' 'Hungary' 'Holand-Netherlands']\n"

发现数据存在occupation缺失而workclass为"Never-worked"的情况，反之则不存在。

这是由于无工作者没有职业，此部分可直接将这些occupation填补为一个新的类即可。

df[['occupation','workclass']][(df['occupation'].isna()==True) & (df['workclass'].isna()==False)]
occupationworkclass
5361NaNNever-worked
10845NaNNever-worked
14772NaNNever-worked
20337NaNNever-worked
23232NaNNever-worked
32304NaNNever-worked
32314NaNNever-worked
df.loc[df['workclass']=='Never_worked','occupation'] = 'Never_worked'

df['workclass'].value_counts()

Private             22696
Self-emp-not-inc     2541
Local-gov            2093
State-gov            1298
Self-emp-inc         1116
Federal-gov           960
Without-pay            14
Never-worked            7
Name: workclass, dtype: int64

df.head()
ageworkclassfinal_weighteducationeducation_nummarital_statusoccupationrelationshipracesexcapital_gaincapital_losshours_per_weeknative_countrysalary
039State-gov77516Bachelors13Never-marriedAdm-clericalNot-in-familyWhiteMale2174040United-States<=50K
150Self-emp-not-inc83311Bachelors13Married-civ-spouseExec-managerialHusbandWhiteMale0013United-States<=50K
238Private215646HS-grad9DivorcedHandlers-cleanersNot-in-familyWhiteMale0040United-States<=50K
353Private23472111th7Married-civ-spouseHandlers-cleanersHusbandBlackMale0040United-States<=50K
428Private338409Bachelors13Married-civ-spouseProf-specialtyWifeBlackFemale0040Cuba<=50K
df.sex.value_counts()

Male      21790
Female    10771
Name: sex, dtype: int64

# 处理性别
df['sex'] = 1*(df['sex']=='Male')

df.head()
ageworkclassfinal_weighteducationeducation_nummarital_statusoccupationrelationshipracesexcapital_gaincapital_losshours_per_weeknative_countrysalary
039State-gov77516Bachelors13Never-marriedAdm-clericalNot-in-familyWhite12174040United-States<=50K
150Self-emp-not-inc83311Bachelors13Married-civ-spouseExec-managerialHusbandWhite10013United-States<=50K
238Private215646HS-grad9DivorcedHandlers-cleanersNot-in-familyWhite10040United-States<=50K
353Private23472111th7Married-civ-spouseHandlers-cleanersHusbandBlack10040United-States<=50K
428Private338409Bachelors13Married-civ-spouseProf-specialtyWifeBlack00040Cuba<=50K
# 删除education
del df['education']

df.head()
ageworkclassfinal_weighteducation_nummarital_statusoccupationrelationshipracesexcapital_gaincapital_losshours_per_weeknative_countrysalary
039State-gov7751613Never-marriedAdm-clericalNot-in-familyWhite12174040United-States<=50K
150Self-emp-not-inc8331113Married-civ-spouseExec-managerialHusbandWhite10013United-States<=50K
238Private2156469DivorcedHandlers-cleanersNot-in-familyWhite10040United-States<=50K
353Private2347217Married-civ-spouseHandlers-cleanersHusbandBlack10040United-States<=50K
428Private33840913Married-civ-spouseProf-specialtyWifeBlack00040Cuba<=50K
# 处理salary 标签 
df['salary'] = 1*(df['salary']=='>50K')

df.salary.value_counts()

0    24720
1     7841
Name: salary, dtype: int64

df.head(5)
ageworkclassfinal_weighteducation_nummarital_statusoccupationrelationshipracesexcapital_gaincapital_losshours_per_weeknative_countrysalary
039State-gov7751613Never-marriedAdm-clericalNot-in-familyWhite12174040United-States0
150Self-emp-not-inc8331113Married-civ-spouseExec-managerialHusbandWhite10013United-States0
238Private2156469DivorcedHandlers-cleanersNot-in-familyWhite10040United-States0
353Private2347217Married-civ-spouseHandlers-cleanersHusbandBlack10040United-States0
428Private33840913Married-civ-spouseProf-specialtyWifeBlack00040Cuba0
temp = df.copy()

# 删除需要独热编码的列，防止列名重复导致编码失败
df.drop(['workclass','marital_status','occupation','relationship','race','native_country'],axis=1,inplace=True)

df.head()
agefinal_weighteducation_numsexcapital_gaincapital_losshours_per_weeksalary
0397751613121740400
1508331113100130
2382156469100400
3532347217100400
42833840913000400
df = df.join(pd.get_dummies(temp.workclass))

df = df.join(pd.get_dummies(temp.marital_status))

df = df.join(pd.get_dummies(temp.occupation))

df = df.join(pd.get_dummies(temp.relationship))
df = df.join(pd.get_dummies(temp.race))
df = df.join(pd.get_dummies(temp.native_country))

df.head()
agefinal_weighteducation_numsexcapital_gaincapital_losshours_per_weeksalaryFederal-govLocal-gov...PortugalPuerto-RicoScotlandSouthTaiwanThailandTrinadad&TobagoUnited-StatesVietnamYugoslavia
039775161312174040000...0000000100
150833111310013000...0000000100
238215646910040000...0000000100
353234721710040000...0000000100
4283384091300040000...0000000000

5 rows × 89 columns

salary = df.pop('salary')
df['salary'] = salary
df.head()
agefinal_weighteducation_numsexcapital_gaincapital_losshours_per_weekFederal-govLocal-govNever-worked...Puerto-RicoScotlandSouthTaiwanThailandTrinadad&TobagoUnited-StatesVietnamYugoslaviasalary
039775161312174040000...0000001000
150833111310013000...0000001000
238215646910040000...0000001000
353234721710040000...0000001000
4283384091300040000...0000000000

5 rows × 89 columns

2.4 划分数据集

# 判断有没有字段还不是数值类型
X = df.iloc[:,:-1].copy()
y = df.iloc[:,-1].copy()
for col_name in X:
    col_data = X[col_name]
    if col_data.dtype == 'object':
        print(col_name,'------->',col_data.unique())

X.head()
agefinal_weighteducation_numsexcapital_gaincapital_losshours_per_weekFederal-govLocal-govNever-worked...PortugalPuerto-RicoScotlandSouthTaiwanThailandTrinadad&TobagoUnited-StatesVietnamYugoslavia
039775161312174040000...0000000100
150833111310013000...0000000100
238215646910040000...0000000100
353234721710040000...0000000100
4283384091300040000...0000000000

5 rows × 88 columns

# 无量纲处理 ：因为已经使用了哑变量，所以此处不需要无量纲再处理一遍了

from sklearn.preprocessing import StandardScaler
s_X = StandardScaler().fit_transform(X)
s_X = pd.DataFrame(data = s_X,columns = X.columns)
s_X.head()
agefinal_weighteducation_numsexcapital_gaincapital_losshours_per_weekFederal-govLocal-govNever-worked...PortugalPuerto-RicoScotlandSouthTaiwanThailandTrinadad&TobagoUnited-StatesVietnamYugoslavia
00.030671-1.0636111.1347390.7030710.148453-0.21666-0.035429-0.174295-0.262097-0.014664...-0.033729-0.059274-0.019201-0.049628-0.039607-0.023518-0.0241630.340954-0.045408-0.022173
10.837109-1.0087071.1347390.703071-0.145920-0.21666-2.222153-0.174295-0.262097-0.014664...-0.033729-0.059274-0.019201-0.049628-0.039607-0.023518-0.0241630.340954-0.045408-0.022173
2-0.0426420.245079-0.4200600.703071-0.145920-0.21666-0.035429-0.174295-0.262097-0.014664...-0.033729-0.059274-0.019201-0.049628-0.039607-0.023518-0.0241630.340954-0.045408-0.022173
31.0570470.425801-1.1974590.703071-0.145920-0.21666-0.035429-0.174295-0.262097-0.014664...-0.033729-0.059274-0.019201-0.049628-0.039607-0.023518-0.0241630.340954-0.045408-0.022173
4-0.7757681.4081761.134739-1.422331-0.145920-0.21666-0.035429-0.174295-0.262097-0.014664...-0.033729-0.059274-0.019201-0.049628-0.039607-0.023518-0.024163-2.932948-0.045408-0.022173

5 rows × 88 columns

from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test = train_test_split(s_X,y)

2.5 训练

1.随机森林

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# 随机森林调参，跑了几个小时才跑完

# RF = RandomForestClassifier()
# param_grid = {'max_features':['auto','sqrt','log2'],
#               'max_depth':np.arange(1,15),
#               'min_samples_leaf':np.arange(1,20),
#               'min_samples_split':[0.1,0.3,0.5],
#               'criterion':['gini','entropy']
#              }
# GS = GridSearchCV(RF,param_grid=param_grid,cv=5,scoring='roc_auc').fit(x_train,y_train)
# print(GS.best_params_)
# print(GS.best_score_)
# print(GS.best_estimator_)

"""
{'criterion': 'gini', 'max_depth': 12, 'max_features': 'auto', 'min_samples_leaf': 1, 'min_samples_split': 0.1}
0.9044572707834764
RandomForestClassifier(max_depth=12, min_samples_split=0.1)
"""

"\n{'criterion': 'gini', 'max_depth': 12, 'max_features': 'auto', 'min_samples_leaf': 1, 'min_samples_split': 0.1}\n0.9044572707834764\nRandomForestClassifier(max_depth=12, min_samples_split=0.1)\n"

RF = RandomForestClassifier(criterion='gini', max_depth = 12, max_features='auto', min_samples_leaf = 3, min_samples_split = 0.1).fit(X_train,y_train)


RF.score(X_test,y_test)

0.844490848790075

from sklearn.metrics import classification_report

print(classification_report(y_test,RF.predict(X_test)))

              precision    recall  f1-score   support

           0       0.85      0.96      0.90      6194
           1       0.79      0.47      0.59      1947

    accuracy                           0.84      8141
   macro avg       0.82      0.72      0.75      8141
weighted avg       0.84      0.84      0.83      8141

​

2 SVM 支持向量机

from sklearn import svm

svc_bal = svm.SVC(class_weight='balanced')  #自动调整不平衡样本
svc_bal.fit(X_train,y_train)

SVC(class_weight='balanced')

svc_bal.score(X_test,y_test)

0.8030954428202923

print(classification_report(y_test,svc_bal.predict(X_test)))

              precision    recall  f1-score   support

           0       0.94      0.80      0.86      6194
           1       0.56      0.83      0.67      1947

    accuracy                           0.80      8141
   macro avg       0.75      0.81      0.76      8141
weighted avg       0.85      0.80      0.81      8141

​

3 逻辑回归

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X_train,y_train)

LogisticRegression()

print(classification_report(y_test,lr.predict(X_test)))

              precision    recall  f1-score   support

           0       0.88      0.93      0.91      6194
           1       0.74      0.61      0.67      1947

    accuracy                           0.86      8141
   macro avg       0.81      0.77      0.79      8141
weighted avg       0.85      0.86      0.85      8141

​

lr.score(X_test,y_test)

0.8560373418498956

from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV

# 网格搜索过程，比较耗时，为方便调试先注释起来
"""
param_grid = {
    'C':[0.01, 0.1,1, 10,20],
    'penalty':['l1','l2'],
    'max_iter':[50, 100, 200, 300,500,1000]
}
gscv = GridSearchCV(estimator=LogisticRegression(),param_grid=param_grid,cv=10)
gscv.fit(ss_X,y)
gscv.best_params_
"""
# 最优参数
# {'C': 0.01, 'max_iter': 50, 'penalty': 'l2'}

"\nparam_grid = {\n    'C':[0.01, 0.1,1, 10,20],\n    'penalty':['l1','l2'],\n    'max_iter':[50, 100, 200, 300,500,1000]\n}\ngscv = GridSearchCV(estimator=LogisticRegression(),param_grid=param_grid,cv=10)\ngscv.fit(ss_X,y)\ngscv.best_params_\n"

# 使用cross_val_score 来进行预测评分
# estimator 算法对象
# X, y = None 特征向量集合 标签集合
# cv Kfold的几折 3 5 10 
lr = LogisticRegression(C=0.01,max_iter=50,penalty='l2')
result = cross_val_score(lr,X,y,cv=10)
result

array([0.79643844, 0.80006143, 0.79361179, 0.79637592, 0.79453317,
       0.80773956, 0.79084767, 0.80128993, 0.79668305, 0.79821867])

result.mean()

0.7975799619643648

4 Adaboost

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

ada = AdaBoostClassifier(
    base_estimator = DecisionTreeClassifier(max_depth=2),
    n_estimators=50,
    learning_rate=1.0
).fit(X_train,y_train)

print(classification_report(y_test,ada.predict(X_test)))

              precision    recall  f1-score   support

           0       0.90      0.94      0.92      6194
           1       0.76      0.66      0.71      1947

    accuracy                           0.87      8141
   macro avg       0.83      0.80      0.81      8141
weighted avg       0.87      0.87      0.87      8141

​

ada.score(X_test,y_test)

0.8700405355607419

5 GDBT

from sklearn.ensemble import GradientBoostingClassifier

gbdt = GradientBoostingClassifier().fit(X_train,y_train)

print(classification_report(y_test,gbdt.predict(X_test)))

              precision    recall  f1-score   support

           0       0.88      0.95      0.92      6194
           1       0.80      0.60      0.69      1947

    accuracy                           0.87      8141
   macro avg       0.84      0.78      0.80      8141
weighted avg       0.86      0.87      0.86      8141

​

6 XGB

from xgboost import XGBClassifier
from xgboost import XGBRegressor

sk_xgb_c = XGBClassifier().fit(X_train,y_train)

print(classification_report(y_test,sk_xgb_c.predict(X_test)))

              precision    recall  f1-score   support

           0       0.90      0.94      0.92      6194
           1       0.77      0.65      0.71      1947

    accuracy                           0.87      8141
   macro avg       0.83      0.80      0.81      8141
weighted avg       0.87      0.87      0.87      8141

​
大地之灯
关注
  • 0
    点赞
  • 踩
  • 4
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
美国人口普查数据及数据说明
07-28
这是Coursera的Intro to Data Analysis for Python课程使用的美国人口普查数据，里面包含了一个数据文件和一个PDF文件，PDF文件解释了每个变量的意义。
python人口普查数据数据分析_美国人口普查数据可视化探索和收入水平预测建模...
weixin_39883906的博客
01-29 2568
数据来源1994年美国人口普查数据。首先在notebook导入包和数据集#导入数据分析包import numpy as npimport pandas as pdimport seaborn as snsimport matplotlib.pyplot as plt%matplotlib inline#导入数据，并查看数据类型df =pd.read_csv('adult.csv')df.head(...
大学python专业-5.Python数据可视化-美国大学各专业的性别差距
weixin_37988176的博客
10-29 148
Visualizing The Gender Gap in College degrees：读取数据：import pandas as pddata=pd.read_csv('./data/percent-bachelors-degrees-women-usa.csv')print(data.columns)#print(data[:2])print(data.describe)可视化：def d...
集成学习--人口普查(美国)收入水平分类预测
人工智能学习生
05-15 533
集成学习--人口普查(美国)收入水平分类预测
Python数据分析与机器学习----收入的预测分析
sunshine543123的博客
06-24 7114
一、题目 利用age、workclass、…、native_country等13个特征预测收入是否超过50k，是一个二分类问题。 二、训练集 32561个样本，每个样本14个特征，其中6个连续性特征、9个离散型特征 三、测试集 16281个样本，每个样本14个特征， 即在测试集中，根据age等14个特征，预测income是否超过50k，二分类问题。 四、说明 部分特征的值为“？”，表示缺失值，需要对其先处理。 五、实验过程和结果 import numpy as np import pandas as pd
机器学习：基于mobilenet 的分类网络
03-12
机器学习
机器学习：efficientnet图像分类的代码
03-18
机器学习
机器学习：efficientNet 分类网络实战代码
最新发布
03-25
机器学习
机器学习KNN分类，葡萄酒分类
05-05
机器学习KNN分类，葡萄酒分类
pandas处理美国人口数据案例.rar
03-18
数据为千锋教育数据分析与人工智能教学视频中涉及的美国人口案例数据CSV文件。可以根据此数据进行实验，帮助大家更好的掌握python中pandas库的使用情况
kaggle-2美国人口普查年收入50K分类
热门推荐
常思考->有目标->重实践->善反思
12-31 2万+
本文主要是使用xgboost， RandomForestClassifier算法实现美国1994年人口普查数据，居民年收入是否超过50K的分类问题。主要内容如下:1 数据预处理 数据信息查看，添加对应的列标签 缺失值处理，以及属性值替换 Ordinal Encoding to Categoricals（string 特征转化为整数编码） 2 模型训练以及验证 xgboost算法分类以及GridS
3 Python数据分析 美国各州人口分析案例 Pandas高级操作 美国大选献金案例 matplotlib
健浩的博客
07-15 4264
Python数据分析 1 案例 人口分析 1.1 数据介绍 数据来源：https://github.com/jakevdp/data-USstates/ 1.1.1 州人口数量表 state-population.csv 字段 字段名 备注 state/region 州/区域 州简称，与state-abbrevs.csv文件中的abbreviation字段对应 ages 人口年龄 选项包括18岁以下under18和全部年龄段total year 年份 population 人口
数据分析实例1（英文报告）--预测未来收入--SAS 逻辑回归--1994年美国人口普查数据
m0_47089011的博客
04-15 4481
Prediction of Future Income by Using Logistic Regression Matthew LaFrance， Yu Zhang 1 Introduction Many factors could influence a person’s annual income, for example, age, gender, race, level of ed...
机器学习入门（1）---程序员薪资数据分析可视化
m0_45056394的博客
05-29 852
话不多说上代码，爬取一些数据进行可视化和机器学习演练 #数据分析库 import pandas as pd #科学计算库 import numpy as np from pandas import Series,DataFrame import matplotlib.pyplot as plt import seaborn as sns plt.rcParams['font.sans-serif'] = ['SimHei'] # 用来正常显示中文标签 plt.rcParams['axes.unicod
数据集：大学毕业生收入
林景的博客
09-24 1万+
数据集：大学毕业生收入 下载地址，本文以绘制直方图为主。 1. 字段描述 字段名称 字段类型 字段说明 Major_code 整型 专业代码。 Major 字符型 专业名称。 Major_category 字符型 专业所属目录。 Total 整型 总人数。 Employed 整型 就业人数。 Employed_full_time_year_round 整型 全年全职在岗人数。 Unemployed 整型 失业人数。 Unemployment_rate 浮点型 失业
【机器学习项目实战】数据相关岗位薪资水平影响因素研究分析
京斗码农的博客
07-21 2467
说明：这是一个机器学习项目全流程（附带项目实例），本篇教程来源于网络，胖哥对此进行了完整的梳理，并把用到的数据+代码完全奉上。如需数据+完整代码可以直接到文章最后获取。 机器学习项目全流程介绍： 一、需求分析 1. 需求来源 个人学习 2. 需求分析 数据科学相关岗位的薪资水平 哪些岗位属于数据科学相关岗位，可以列出多个关键字：数据分析师、数据挖掘工程师、大数据工程师、机器学习算法工程师、深度学习算法工程师等 二、数据采集 爬取拉勾网数据,本次学习所需数据已爬取好，数据量500条左右...
【笔记】数据分析实战：从数据清洗到ROC曲线
SpriteNym的博客
08-05 210
很实用
【DL】第 2 章 ： 深度学习的最佳介绍
sikh_0529的博客
02-25 392
在下一节深度学习教程介绍中，我们将介绍深度学习的必要性和重要性。在我们深入了解深度学习及其应用程序和平台之前，这篇深度学习介绍教程将首先帮助您了解什么是深度学习。深度学习是机器学习的一个子领域，它处理受大脑结构和功能启发的算法。输出是人工神经元预测的最终值。深度学习是机器学习的一种应用，它使用复杂的算法和深度神经网络来训练模型。现在您已经知道深度学习到底是什么，它的应用和重要性，接下来在深度学习教程的介绍中，让我们看看神经网络及其操作。如您所见，我们的实际输入是一个正方形，但神经网络预测的输出是一个圆形。
机器学习：逻辑回归的算法原理和使用场景
11-03
逻辑回归是机器学习中的一种分类模型，它是一种分类算法，虽然名字中带有回归。逻辑回归的算法原理是通过对数据进行拟合，得到一个能够将数据分为不同类别的函数，这个函数被称为sigmoid函数。逻辑回归的使用场景非常广泛，例如广告点击率、是否为垃圾邮件、是否患病、金融诈骗、虚假账号等。 逻辑回归的算法原理： 1. 逻辑回归是一种广义线性模型，它的基本思想是通过对数据进行拟合，得到一个能够将数据分为不同类别的函数，这个函数被称为sigmoid函数。 2. sigmoid函数的形式为：$g(z)=\frac{1}{1+e^{-z}}$，其中z为输入的线性函数，$e$为自然常数。 3. 在逻辑回归中，我们需要通过最大化似然函数来确定模型的参数，从而得到最优的分类结果。 逻辑回归的使用场景： 1. 广告点击率预测：通过对用户的历史点击数据进行分析，预测用户是否会点击某个广告。 2. 垃圾邮件分类：通过对邮件的文本内容进行分析，判断邮件是否为垃圾邮件。 3. 疾病诊断：通过对患者的病历数据进行分析，判断患者是否患有某种疾病。 4. 金融诈骗检测：通过对用户的交易数据进行分析，判断是否存在金融诈骗行为。 5. 虚假账号检测：通过对用户的行为数据进行分析，判断是否存在虚假账号。

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值