scrapy框架学习-爬取腾讯社招信息-item字段和管道文件

最新推荐文章于 2023-09-20 15:38:22 发布

小明的博客

最新推荐文章于 2023-09-20 15:38:22 发布

阅读量903

点赞数

分类专栏： python学习笔记文章标签：框架

本文链接：https://blog.csdn.net/sb_Ihateyou/article/details/78885415

版权

python学习笔记专栏收录该内容

16 篇文章 0 订阅

订阅专栏

本文介绍了如何利用Scrapy框架来爬取腾讯的社招信息，详细讲解了item定义和pipeline的运用，以实现数据的有效抓取和处理。

摘要由CSDN通过智能技术生成

item

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class TenxunItem(scrapy.Item):
    # define the fields for your item here like:
    job_name = scrapy.Field()

    job_link = scrapy.Field()

    job_type = scrapy.Field()

    job_people_num = scrapy.Field()

    job_site = scrapy.Field()

    publish_time = scrapy.Field()

pipelines

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html


import json


class TenxunPipeline(object):
    def __init__(self):
        self.f = open("tencent.json", "w")

    def process_item(self, item, spider):
        content = json.dumps(dict(item),
                             ensure_ascii=False) + ",\n"                                                             "\n"
        self.f.write(content)
        return item

    def close_spider(self, spider):
        self.f.close()