1. 创建项目:scrapy startproject scrapy_test
2. 进入项目目录:cd scrapy_test
3. 生成spiders文件:scrapy genspider baidu baidu.com (scrapy genspider + 名称 + 域名)
4. 新建虚拟环境 : settings -> project -> python interpreter -> Add interpreter ->(默认在本目录下创建一个venv)。打开终端,出现 (venv), 说明虚拟环境已经激活 例如:(venv) PS D:\test_code\scr\test_t1>
5. 在虚拟环境下安装 scrapy pip install scrapy -i https://pypi.tuna.tsinghua.edu.cn/simple
6. 测试代码:
baidu.py:
import scrapy
from scrapy import Selector
from scrapy_test.items import ScrapyTestItem
class BaiduSpider(scrapy.Spider):
name = "baidu"
allowed_domains = ["baidu.com"]
start_urls = ["https://top.baidu.com/board?tab=realtime"]
def parse(self, response):
sel = Selector(response)
l1 = sel.css('#sanRoot > main > div.container.right-container_2EFJr > div > div > div > div.content_1YWBm > a > div.c-single-text-ellipsis::text').extract()
for i in l1:
news = ScrapyTestItem()
news['news'] = i
yield news
items.py:
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class ScrapyTestItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
news = scrapy.Field()
settings.py 添加 USER_AGENT
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
# 放开注释
ITEM_PIPELINES = {
"scrapy_test.pipelines.ScrapyTestPipeline": 300,
}
pipelines.py
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# useful for handling different item types with a single interface
from itemadapter import ItemAdapter
import openpyxl
class ScrapyTestPipeline:
def __init__(self):
self.wb = openpyxl.Workbook()
self.ws = self.wb.active
self.ws.title = 'news_hot'
self.ws.append(('news_titile', 'test'))
def close_spider(self, spider):
self.wb.save('news_hot.xlsx')
def process_item(self, item, spider):
news = item.get('news', '')
self.ws.append((news, ''))
return item
7. 生成 .csv 文件: scrapy crawl baidu -o test.csv
生成 excel 文件: scrapy crawl baidu