Scrapy 基本用法

1. 创建项目：scrapy startproject scrapy_test

2. 进入项目目录：cd scrapy_test

3. 生成spiders文件：scrapy genspider baidu baidu.com  (scrapy genspider + 名称 + 域名)

4. 新建虚拟环境 ： settings -> project -> python interpreter -> Add interpreter ->(默认在本目录下创建一个venv)。打开终端，出现 (venv)， 说明虚拟环境已经激活 例如：(venv) PS D:\test_code\scr\test_t1>

5. 在虚拟环境下安装 scrapy    pip install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple

6. 测试代码：

baidu.py:

import scrapy
from scrapy import Selector
from scrapy_test.items import ScrapyTestItem

class BaiduSpider(scrapy.Spider):
    name = "baidu"
    allowed_domains = ["baidu.com"]
    start_urls = ["https://top.baidu.com/board?tab=realtime"]

    def parse(self, response):
        sel = Selector(response)
        l1 = sel.css('#sanRoot > main > div.container.right-container_2EFJr > div > div > div > div.content_1YWBm > a > div.c-single-text-ellipsis::text').extract()

        for i in l1:
            news = ScrapyTestItem()
            news['news'] = i

            yield news

items.py:

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class ScrapyTestItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    news = scrapy.Field()

settings.py 添加 USER_AGENT

USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"



# 放开注释
ITEM_PIPELINES = {
   "scrapy_test.pipelines.ScrapyTestPipeline": 300,
}

pipelines.py

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
import openpyxl

class ScrapyTestPipeline:

    def __init__(self):
        self.wb = openpyxl.Workbook()
        self.ws = self.wb.active
        self.ws.title = 'news_hot'
        self.ws.append(('news_titile', 'test'))

    def close_spider(self, spider):
        self.wb.save('news_hot.xlsx')

    def process_item(self, item, spider):
        news = item.get('news', '')
        self.ws.append((news, ''))
        return item

7. 生成 .csv 文件： scrapy crawl baidu -o test.csv

    生成 excel 文件： scrapy crawl baidu