BS1048-基于Python+scrapy+Echarts开发实现互联网职位画像分析系统

本基于Python+scrapy+Echarts互联网职位画像分析系统，系统采用多层MVC软件架构，采用Python Django scrapy框架实现互联网职位的爬虫采集程序，数据分析程序以及数据可视化平台。系统爬虫端主要采用scrapy实时采集分析智联招聘、boss直聘等工作网站岗位数据，
通过数据过滤，去重，变换成规则化的数据写入mysql数据库。系统服务端主要提取岗位数据进行关联性、计算性分析，提供数据可视化端程序调用。数据可视化平台主要采用echarts图表技术，实现针对岗位数据进行画像处理，分析不同维度的岗位热度等等。

原文地址

一、程序设计

本次基于Python+scrapy+Echarts互联网职位画像分析系统，主要内容涉及：

主要功能模块：书籍聚类图谱，书籍专业统计，书籍文献管理，相似书籍推荐，热门书籍排行，我的借阅，系统管理等等
主要包含技术：Python，scrapy，Django，Echarts，网络爬虫，mysql，html，javascript，echarts，vue.js
主要包含算法：数据分析计算等

二、效果实现

职位月薪与学历

月薪与公司性质

职位月薪与地点

月薪与招聘人数

其他效果省略

三、核心代码

1.岗位数据可视化
本系统岗位数据可视化模块，主要采用Python搭建后台计算接口，分析mysql数据库中用户的所有岗位信息，统计系统内部的招聘岗位月薪按照时间的分布情况，给出柱状图及折线图、地图，饼状图等多种方式分析不同岗位在各时间段的招聘情况。

# 地点(地图+柱状)
def gen_zwyx_dd(zwlb):
    qs = ZpZwByAreaModel.objects
    if zwlb:
        qs = qs.filter(zwlb=zwlb)
        path=f'zwyx_dd/{zwlb}.html'
    else:
        path = 'zwyx_dd.html'
    page=Page()
    df = read_frame(qs.all())
    if len(df)>0:
        df_group=df.groupby(['year','month'])
        time_line_chart1= Timeline(width=1500, height=450,is_auto_play=False,timeline_bottom=0)
        time_line_chart2= Timeline(width=1500, height=450,is_auto_play=False,timeline_bottom=0)
        for name,group in df_group:
            # 地图 平均薪资
            month=group['month'].tolist()[0]
            year=group['year'].tolist()[0]
            df_new=group.groupby('province').apply(get_echarts_all_by_zwyx_value,'province')
            data = [(a, (b + c) / 2) for a, b, c in
                     zip(df_new['province'].tolist(), df_new['max_zwyx'].tolist(), df_new['min_zwyx'].tolist())]
            chart = Map(f'{zwlb}平均职位月薪与地点',width=1500, height=450)
            attr, value = chart.cast(data)
            chart.add(f'平均薪资', attr, value, wmaptype='china', is_label_show=True, is_visualmap=True,
                       visual_range=[int(min(value)), int(max(value))],visual_pos='right',visual_top='top')
            time_line_chart1.add(chart,f'{year}年{month}月')
        # 本月职位量<span class="token class-name">Top20</span>
        chart3<span class="token operator">=</span><span class="token class-name">Pie</span><span class="token punctuation">(</span>f'<span class="token punctuation">{<!-- --></span>zwlb<span class="token punctuation">}</span>职位量及招聘人数'<span class="token punctuation">,</span>width<span class="token operator">=</span><span class="token number">1500</span><span class="token punctuation">)</span>
        chart3<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token char">'职位量'</span><span class="token punctuation">,</span> df_new<span class="token punctuation">[</span>'province'<span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">tolist</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> df_new<span class="token punctuation">[</span><span class="token char">'count'</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">tolist</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>center<span class="token operator">=</span><span class="token punctuation">[</span><span class="token number">25</span><span class="token punctuation">,</span><span class="token number">50</span><span class="token punctuation">]</span><span class="token punctuation">,</span>is_label_show<span class="token operator">=</span><span class="token class-name">True</span><span class="token punctuation">)</span>
        chart3<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token char">'招聘人数'</span><span class="token punctuation">,</span> df_new<span class="token punctuation">[</span>'province'<span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">tolist</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> df_new<span class="token punctuation">[</span><span class="token char">'zprs'</span><span class="token punctuation">]</span><span class="token punctuation">.</span><span class="token function">tolist</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span>center<span class="token operator">=</span><span class="token punctuation">[</span><span class="token number">75</span><span class="token punctuation">,</span><span class="token number">50</span><span class="token punctuation">]</span><span class="token punctuation">,</span>is_label_show<span class="token operator">=</span><span class="token class-name">True</span><span class="token punctuation">)</span>
        time_line_chart2<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span>chart3<span class="token punctuation">,</span>f'<span class="token punctuation">{<!-- --></span>year<span class="token punctuation">}</span>年<span class="token punctuation">{<!-- --></span>month<span class="token punctuation">}</span>月'<span class="token punctuation">)</span>
    page<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span>time_line_chart1<span class="token punctuation">)</span>
    page<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span>time_line_chart2<span class="token punctuation">)</span>
    page<span class="token punctuation">.</span><span class="token function">render</span><span class="token punctuation">(</span>os<span class="token punctuation">.</span>path<span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span>BASE_DIR<span class="token punctuation">,</span> 'templates<span class="token operator">/</span><span class="token punctuation">{<!-- --></span><span class="token punctuation">}</span>'<span class="token punctuation">.</span><span class="token function">format</span><span class="token punctuation">(</span>path<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">)</span>