统计某招聘网岗位职责要求关键字权重

继上一篇D3.js - 某招聘网信息统计可视化简单统计了一些职位和公司比例信息，有时候某些简历自动分拣系统等对关键字进行过滤，这时就有必要去统计一下某些职位下关键字的权重。

准备数据

这里取职位描述内容进行关键词提取，一次性从数据库提取匹配记录汇总到一个文本文件，直接对标题进行like匹配，这里以“世上最好的语言PHP”为例

select title, content from job where title like "%php%";

写到一个文件

print(len(res))
with open('php.txt', 'wt') as f:
    for r in res:
        f.write(r['content'])

分词统计

这里使用Python版结巴分词jieba中基于TF-IDF算法的关键词抽取进行权重计算；相对统计数量，权重显得更直观一点。

这里使用自带的demo extract_tags_with_weight.py就行了

取前十关键字权重（34882条PHP招聘信息）

python3 extract_tags_with_weight.py php.txt -k 10 -w 1

这里有中文，这些并不怎么需要，过滤掉中文取听top10技术关键字

python3 extract_tags_with_weight.py php.txt -k 999999 -w 1 | grep 'tag: [a-z]' | sed -n '1,10p'

这里把-k参数调最大取全部，同时也要将文本全转成小写或大写防止重复

其他职位

权重是全部关键字统计所得，已去除中文部分

php top 50

记录：34882

tag: php                  weight: 0.241653
tag: mysql                weight: 0.127151
tag: web                  weight: 0.070640
tag: linux                weight: 0.062477
tag: css                  weight: 0.061224
tag: javascript           weight: 0.060166
tag: html                 weight: 0.052278
tag: ajax                 weight: 0.040207
tag: jquery               weight: 0.033997
tag: mvc                  weight: 0.024620
tag: sql                  weight: 0.024506
tag: thinkphp             weight: 0.021231
tag: lamp                 weight: 0.020800
tag: xml                  weight: 0.020014
tag: div                  weight: 0.018878
tag: redis                weight: 0.018336
tag: apache               weight: 0.017596
tag: js                   weight: 0.016749
tag: yii                  weight: 0.015015
tag: nginx                weight: 0.014331
tag: html5                weight: 0.011799
tag: nosql                weight: 0.011088
tag: xhtml                weight: 0.010838
tag: app                  weight: 0.010734
tag: http                 weight: 0.010595
tag: unix                 weight: 0.010556
tag: shell                weight: 0.010319
tag: json                 weight: 0.010156
tag: memcache             weight: 0.010130
tag: oop                  weight: 0.010030
tag: lnmp                 weight: 0.009845
tag: java                 weight: 0.008475
tag: ci                   weight: 0.008348
tag: mongodb              weight: 0.007690
tag: smarty               weight: 0.007537
tag: api                  weight: 0.007303
tag: python               weight: 0.007112
tag: memcached            weight: 0.007014
tag: zend                 weight: 0.006881
tag: svn                  weight: 0.006774
tag: ecshop               weight: 0.006349
tag: bug                  weight: 0.006180
tag: git                  weight: 0.005200
tag: discuz               weight: 0.005051
tag: oracle               weight: 0.004645
tag: w3c                  weight: 0.004577
tag: c++                  weight: 0.004544
tag: cms                  weight: 0.004515
tag: framework            weight: 0.004012
tag: css3                 weight: 0.003905

python top 50

记录：3229

tag: python               weight: 0.217281
tag: web                  weight: 0.107951
tag: linux                weight: 0.085868
tag: mysql                weight: 0.061585
tag: django               weight: 0.060428
tag: redis                weight: 0.030252
tag: javascript           weight: 0.029916
tag: css                  weight: 0.029580
tag: html                 weight: 0.028722
tag: tornado              weight: 0.028125
tag: mongodb              weight: 0.027454
tag: flask                weight: 0.024022
tag: java                 weight: 0.021075
tag: git                  weight: 0.020255
tag: http                 weight: 0.019471
tag: php                  weight: 0.017905
tag: shell                weight: 0.016562
tag: sql                  weight: 0.015741
tag: nginx                weight: 0.015480
tag: c++                  weight: 0.014809
tag: api                  weight: 0.013503
tag: jquery               weight: 0.012981
tag: nosql                weight: 0.012086
tag: js                   weight: 0.011713
tag: mvc                  weight: 0.011228
tag: postgresql           weight: 0.010668
tag: app                  weight: 0.010594
tag: github               weight: 0.009586
tag: tcp                  weight: 0.009437
tag: ip                   weight: 0.008691
tag: html5                weight: 0.007908
tag: unix                 weight: 0.007460
tag: ruby                 weight: 0.007199
tag: restful              weight: 0.007087
tag: apache               weight: 0.006267
tag: ajax                 weight: 0.006043
tag: py                   weight: 0.006006
tag: openstack            weight: 0.005968
tag: hadoop               weight: 0.005670
tag: mac                  weight: 0.005670
tag: memcache             weight: 0.005483
tag: com                  weight: 0.005260
tag: socket               weight: 0.005036
tag: xml                  weight: 0.004961
tag: wecash               weight: 0.004887
tag: oracle               weight: 0.004513
tag: www                  weight: 0.004103
tag: svn                  weight: 0.003991
tag: experience           weight: 0.003991
tag: bug                  weight: 0.003954

java top 50

记录：53599

tag: java                 weight: 0.173577
tag: web                  weight: 0.069864
tag: spring               weight: 0.069113
tag: mysql                weight: 0.065630
tag: oracle               weight: 0.048071
tag: j2ee                 weight: 0.046025
tag: sql                  weight: 0.045887
tag: javascript           weight: 0.045806
tag: linux                weight: 0.040972
tag: hibernate            weight: 0.038958
tag: jquery               weight: 0.033819
tag: html                 weight: 0.032508
tag: tomcat               weight: 0.031317
tag: css                  weight: 0.029937
tag: struts               weight: 0.028279
tag: ajax                 weight: 0.026473
tag: jsp                  weight: 0.022588
tag: mvc                  weight: 0.020513
tag: ibatis               weight: 0.020188
tag: mybatis              weight: 0.019560
tag: servlet              weight: 0.015777
tag: xml                  weight: 0.015701
tag: js                   weight: 0.014403
tag: redis                weight: 0.013724
tag: eclipse              weight: 0.013452
tag: weblogic             weight: 0.012535
tag: springmvc            weight: 0.011370
tag: jboss                weight: 0.011296
tag: struts2              weight: 0.011183
tag: http                 weight: 0.010566
tag: server               weight: 0.010471
tag: svn                  weight: 0.009402
tag: ssh                  weight: 0.009260
tag: nosql                weight: 0.008990
tag: sqlserver            weight: 0.008626
tag: html5                weight: 0.008624
tag: uml                  weight: 0.008324
tag: apache               weight: 0.008202
tag: maven                weight: 0.008192
tag: app                  weight: 0.007383
tag: mongodb              weight: 0.007200
tag: shell                weight: 0.007161
tag: nginx                weight: 0.007028
tag: jdbc                 weight: 0.006908
tag: unix                 weight: 0.006865
tag: json                 weight: 0.006723
tag: webservice           weight: 0.006723
tag: tcp                  weight: 0.006310
tag: websphere            weight: 0.006182
tag: io                   weight: 0.005851

android top 50

记录：33434

tag: android              weight: 0.477328
tag: java                 weight: 0.092136
tag: app                  weight: 0.057003
tag: ui                   weight: 0.047871
tag: sdk                  weight: 0.039509
tag: http                 weight: 0.037914
tag: tcp                  weight: 0.025357
tag: socket               weight: 0.024767
tag: ip                   weight: 0.020937
tag: c++                  weight: 0.020032
tag: ios                  weight: 0.019612
tag: xml                  weight: 0.015327
tag: api                  weight: 0.014370
tag: json                 weight: 0.013632
tag: framework            weight: 0.012522
tag: linux                weight: 0.012276
tag: eclipse              weight: 0.009863
tag: ndk                  weight: 0.009170
tag: sqlite               weight: 0.007842
tag: html5                weight: 0.007360
tag: web                  weight: 0.007267
tag: jni                  weight: 0.005627
tag: andriod              weight: 0.005464
tag: bug                  weight: 0.005176
tag: os                   weight: 0.004833
tag: svn                  weight: 0.004469
tag: git                  weight: 0.004406
tag: gui                  weight: 0.003779
tag: github               weight: 0.003699
tag: javascript           weight: 0.003682
tag: https                weight: 0.003599
tag: sql                  weight: 0.003401
tag: html                 weight: 0.003359
tag: com                  weight: 0.003169
tag: udp                  weight: 0.003079
tag: service              weight: 0.003075
tag: mysql                weight: 0.002957
tag: wifi                 weight: 0.002829
tag: j2me                 weight: 0.002683
tag: studio               weight: 0.002631
tag: www                  weight: 0.002548
tag: mvc                  weight: 0.002517
tag: google               weight: 0.002309
tag: js                   weight: 0.002305
tag: lbs                  weight: 0.002257
tag: im                   weight: 0.002233
tag: objective            weight: 0.002177
tag: server               weight: 0.002125
tag: activity             weight: 0.002118
tag: iphone               weight: 0.002115

ios top 50

记录：27632

tag: ios                  weight: 0.366072
tag: app                  weight: 0.084167
tag: objective            weight: 0.071731
tag: iphone               weight: 0.069397
tag: sdk                  weight: 0.051040
tag: xcode                weight: 0.047470
tag: c++                  weight: 0.045456
tag: ui                   weight: 0.042223
tag: http                 weight: 0.039140
tag: ipad                 weight: 0.032945
tag: android              weight: 0.025376
tag: tcp                  weight: 0.021865
tag: socket               weight: 0.021082
tag: json                 weight: 0.020783
tag: xml                  weight: 0.020325
tag: store                weight: 0.019605
tag: mac                  weight: 0.018228
tag: cocoa                weight: 0.016381
tag: ip                   weight: 0.015349
tag: object               weight: 0.014816
tag: os                   weight: 0.012732
tag: appstore             weight: 0.011205
tag: java                 weight: 0.010568
tag: swift                weight: 0.009782
tag: web                  weight: 0.008317
tag: api                  weight: 0.007768
tag: touch                weight: 0.007061
tag: interface            weight: 0.006853
tag: builder              weight: 0.006794
tag: html5                weight: 0.006774
tag: sqlite               weight: 0.006387
tag: gui                  weight: 0.006187
tag: mvc                  weight: 0.005925
tag: uikit                weight: 0.005908
tag: bug                  weight: 0.005272
tag: github               weight: 0.005055
tag: udp                  weight: 0.004772
tag: instruments          weight: 0.004752
tag: core                 weight: 0.004327
tag: ue                   weight: 0.004152
tag: git                  weight: 0.004057
tag: framework            weight: 0.003807
tag: macos                weight: 0.003582
tag: service              weight: 0.003483
tag: runtime              weight: 0.003233
tag: javascript           weight: 0.003037
tag: com                  weight: 0.003025
tag: im                   weight: 0.002896
tag: apple                weight: 0.002863
tag: oop                  weight: 0.002642