【scrapy 段子网站的内容爬取】

创建项目

scrapy startproject xiaohua

进入项目

 cd .\xiaohua\

创建文件

scrapy genspider duanzi www.a.com

setting.py 设置

添加
LOG_LEVEL = "ERROR"
true变false
ROBOTSTXT_OBEY = False
启用useragent
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"

编写爬虫文件

获取url
https://www.lehuoba.com/gaoxiaoduanzi/

start_urls = ["https://www.lehuoba.com/gaoxiaoduanzi/"]

解析数据

 ele_list = response.xpath('/html/body/section/div[1]/div/article')
 for ele in ele_list:
      detail_url = ele.xpath("./header/h2/a/@href").extract_first()
      title = ele.xpath("./header/h2/a//text()").extract_first()

创建item

class XiaohuaItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    content = scrapy.Field()
    pass

使用item

from xiaohua.items import XiaohuaItem

item = XiaohuaItem()
item['title'] = title

实现代码

    def detail_parse(self, response):
        item = response.meta['item']
        # 获取详情页数据
        content = response.xpath('/html/body/section/div[1]/div/article//text()').extract_first()
        item['content'] = content
        
        yield item

    def parse(self, response):
        ele_list = response.xpath('/html/body/section/div[1]/div/article')
        for ele in ele_list:
            detail_url = ele.xpath("./header/h2/a/@href").extract_first()
            title = ele.xpath("./header/h2/a//text()").extract_first()

            item = XiaohuaItem()
            item['title'] = title

        #     根据url 去请求详情页
            yield scrapy.Request(detail_url, callback=self.detail_parse, meta={'item': item})

yield scrapy.Request(detail_url, callback=self.detail_parse, meta={'item': item})

执行详情页的url爬取  爬取后的数据给detail_parse  顺带把meta参数传进去


def detail_parse(self, response):

接受爬取的页面数据 和meta参数


 yield item
启用piplines功能

使用piplines 存储数据

class XiaohuaPipeline:
    fs = None

    # 重写父类方法
    def open_spider(self, spider):
        print("开始爬取")
        self.fs = open("./xiaohua.txt", 'w+', encoding='utf-8')

    def process_item(self, item, spider):
        # 提取数据写入
        title = item['title']
        content = item['content']

        self.fs.write(title + content + '\n')
        return item


    def close_spider(self, spider):
        # 关闭文件
        self.fs.close()
        print("爬取结束")

setting.py中开启

ITEM_PIPELINES = {
   "xiaohua.pipelines.XiaohuaPipeline": 300,
}

运行代码

 scrapy crawl duanzi

结果写入文件

总结图