python–scrapy框架问题
问题:在使用scrapy框架时,继承ImagesPipeline的Pipeline管道类无法使用,该继承类中重写的的方法无法使用
1.item字段
item.py
import scrapy
class AaproItem(scrapy.Item):
name=scrapy.Field()
src=scrapy.Field()
2.settings配置
UA伪装:
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3947.100 Safari/537.36'
君子协定:
ROBOTSTXT_OBEY =False
日志只输出报错内容:
LOG_LEVEL='ERROR'
启用Item Pipeline:
ITEM_PIPELINES = {
'AaPro.pipelines.AaproPipeline': 300,
}
3.spider爬虫
myspider.py
import scrapy
from AaPro.items import AaproItem
class MyspiderSpider(scrapy.Spider):
name = 'myspider'
start_urls = ['http://www.521609.com/daxuexiaohua/']
def parse(self, response):
li_list=response.xpath('//*[@id="content"]/div[2]/div[2]/ul/li')
for li in li_list:
name=