使用scrapy框架的爬取腾讯招聘的完整代码

最新推荐文章于 2023-11-12 10:20:02 发布

一个回和

最新推荐文章于 2023-11-12 10:20:02 发布

阅读量1.9k

点赞数 2

分类专栏： Scrapy框架高级爬虫项目文章标签： scrapy

本文链接：https://blog.csdn.net/g_optimistic/article/details/90058038

版权

创建项目：

scrapy startproject scrapyProject

创建项目下的小爬虫：

scrapy genspider s_tencent careers.tencent.com

1.spiders文件夹下的s_tencent.py

1.spiders文件夹下的s_tencent.py

# -*- coding: utf-8 -*-
import scrapy
import json
from scrapyProject.items import ScrapyprojectItem


class STencentSpider(scrapy.Spider):
    name = 's_tencent'
    allowed_domains = ['careers.tencent.com']
    start_urls = []
    for page in range(1, 62):
        url = 'https://careers.tencent.com/tencentcareer/api/post/Query?keyword=python&pageIndex=%s&pageSize=10' % page
        start_urls.append(url)

    def parse(self, response):
        # 读response的页面信息
        content = response.body.decode('utf-8')
        # json