爬虫scrapy框架 爬取一药网

cmd 命令创建项目

scrapy startproject yiyaowang
cd yiyaowang
scrapy genspider yaowang yaowang.com

先进入settings.py文件将服从爬虫协议改成False，因为有些网站不盖爬取不了，因此都改了

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class YiyaowangItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()

    # 定义药品名
    title = scrapy.Field()
    # 定义药品价格
    price = scrapy.Field()
    # 定义评论数
    comment = scrapy.Field()
    pass

yaowang.py

# -*- coding: utf-8 -*-
import scrapy
from ..items import YiyaowangItem

class YaowangSp