python爬取数据搭建Hadoop、Flume用Hive做数据分析Sqoop存储到Mysql并实现可视化

最新推荐文章于 2023-07-04 21:03:55 发布

siena2077

最新推荐文章于 2023-07-04 21:03:55 发布

阅读量3.4k

点赞数 6

文章标签：大数据 hive

本文链接：https://blog.csdn.net/siena2077/article/details/107357744

版权

本文介绍了使用Python爬虫获取中华英才网和前程无忧的职位数据，通过Hadoop、Flume进行数据传输，利用Hive进行数据分析， Sqoop将结果导入Mysql并实现数据可视化。具体步骤包括数据爬取、数据存储、Flume配置、Hive数据库操作、Sqoop导出数据以及在MySQL中的数据可视化呈现。

摘要由CSDN通过智能技术生成

加粗样式1.爬虫
爬取的是中华英才网和前程无忧两个网站。

spider代码：

# -*- coding: utf-8 -*-
import copy
import re
import pandas as pd
from scrapy import Spider, Request
from qiancheng.items import QianchengItem


class A51jobSpider(Spider):
    name = '51job'
    # allowed_domains = ['www.51job.com/']
    start_urls = ['https://search.51job.com/list/000000,000000,0000,00,9,99,%25E5%25A4%25A7%25E6%2595%25B0%25E6%258D%25AE%25E5%25BC%2580%25E5%258F%2591%25E5%25B7%25A5%25E7%25A8%258B%25E5%25B8%2588,2,{0}.html?lang=c&stype=&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&providesalary=99&lonlat=0%2C0&radius=-1&ord_field=0&confirmdate=9&fromType=&dibiaoid=0&address=&line=&specialarea=00&from=&welfare='.format(num)for num in range(1, 2000)]

    def parse(self, response):
        a_list = response.xpath('//div[@class="dw_table"]//div[@class="el"]')
        # print(a_list)
        for a in a_list:
            item = QianchengItem()
            # print(a)
            item['work_name'] = a.xpath('//p/span/a/text()').extract_first()
            item['work_name'] = [str(i).strip() for i in item['work_name']]
            item['work_name'] = ''.join(item['work_name'])
            # print(item['work_name'])
            item['company'] = a.xpath('//span[@class="t2"]/a/text()').extract_first()
            item['place'] = a.xpath('./span[@class="t3"]/text()').extract_first()
            item['salary'] = a

最低0.47元/天解锁文章