Scrapy 爬虫框架01—— logging（日志）

最新推荐文章于 2024-04-27 12:58:57 发布
最新推荐文章于 2024-04-27 12:58:57 发布
阅读量755 收藏 1
点赞数
分类专栏： Scarpy 文章标签： Scrapy Log
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/magicboom/article/details/89642168
版权

1.为什么要使用Logging?

  在进行测试的时候，许多初学Pythoner喜欢使用Print函数，这样本身没什么不好，

但是在使用scrapy的时候，由于Scrapy 是多线程，所以你有时候没有办法确定在terminal上显示的data到底是哪里的data.

这时，我们可以使用logging 模块。

 

首先看在scrapy中使用logging模块。

首先是看scrapy 文件：

# -*- coding: utf-8 -*-
import scrapy
import  logging

logger  = logging.getLogger(__name__)

class ItcastSpider(scrapy.Spider):
    name = "itcast"   #  爬虫名
    allowed_domains = ["itcast.cn"]    #允许爬取的范围
    start_urls = ['http://www.itcast.cn/channel/teacher.shtml']  #最开始请求url地址

    def parse(self, response):

        data = response.xpath("//div[@class='tea_con']//li")
        # 分组组成一个item字典
        for li in data:
            item1 = {}
            item1['name'] = li.xpath(".//h3/text()").extract_first()
            item1['title']= li.xpath('.//h4/text()').extract_first()
            logger.warning('this is spider!!')
            yield item1

 logger.warning(item1)   #  把logger中warning的部分output

这里是根据您的settings.py 里面的 LOG_LEVEL确定的

LOG_LEVEL='WARNING'

ok，我们也可以在pipelines里面写，这样就能看清楚数据被哪个py文件处理。

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import  logging
logger = logging.getLogger(__name__)

class Myspiderpro1Pipeline(object):
    def process_item(self, item, spider):
        if spider.name=='itcast':
            logger.warning('this is pipelines!!')
            pass

        return item


class Myspiderpro1Pipeline2(object):
    def process_item(self, item, spider):
        item['rank']="A"
        return item

执行爬虫：

scrapy crawl itcast

2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:32 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:32 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!
2019-04-28 17:33:33 [myspiderpro1.spiders.itcast] WARNING: this is spider!!
2019-04-28 17:33:33 [myspiderpro1.pipelines] WARNING: this is pipelines!!

这么长的logger..很清楚是吧。。

当然我们可以把日志保存。

在settings.py中进行设置：（LOG_FILE=filename）

LOG_LEVEL='WARNING'
LOG_FILE='./log1.log'

再次执行，就不会有长输出而是多了log 文件：

通过看log 文件就可以知道爬虫的进度与效果。

Luke Liu
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
logging模块的使用1——在scrapy框架中
yuhui_2000的博客
10-22 304
简单设置log 1.LOG_LEVEL=“WARN” 为了让我们自己希望输出到终端的内容能够更容易看懂一些“ 我们可以在settings.py中设置log的级别 在settings.py中添加一行（全部大写） LOG_LEVEL="WARN" 这样的话，默认终端显示的就是比"WARN"等级要高的log信息 添加之前终端的显示 启动爬虫：scrapy crawl itcast 2020-10-22 15:45:37 [scrapy.utils.log] INFO: Scrapy 2.4.0 start
【Scrapy爬虫框架】{0} ——Scrapy爬虫框架介绍
12-22
Scrapy是一个功能强大并且非常快速的网络爬虫框架，是非常优秀的python第三方库，也是基于python实现网络爬虫的重要的技术路线。 Scrapy的安装： 直接在命令提示符窗口执行pip install scrapy貌似不行。 我们需要先...
scrapy中logging使用
ITMAN009的博客
05-05 453
LoggingScrapy提供了log功能。您可以通过 scrapy.log 模块使用。当前底层实现使用了 Twisted logging ，不过可能在之后会有所变化。 log服务必须通过显示调用 scrapy.log.start() 来开启。 Log levels¶ Scrapy提供5层logging级别: CRITICAL - 严重错误(critical) ERROR - 一般错误(...
scrapy的logging设置
weixin_42628986的博客
10-09 1849
个人代码 spider代码（见else） layNum = 0 for lay in info: layNum += 1 layoutID = self.newspaperID + publishDate + '{:0>2}'.format(layNum) itemLay...
【Python从入门到进阶】53、Scrapy日志信息及日志级别
最新发布
程序猿之洞
04-27 1081
在Scrapy框架中，日志扮演着至关重要的角色。日志不仅记录了爬虫在运行过程中的详细行为，还提供了大量有助于开发者定位问题和优化程序的信息。通过日志，开发者可以了解到爬虫的每一步操作，包括请求的发送、响应的接收、数据的解析以及任何可能出现的异常。这些信息对于调试程序、监控运行状态以及优化性能都是不可或缺的。
第1.6章 scrapy之logger
warrah 南极狼
06-14 1550
网上有很多介绍logger的配置，但是我在执行过程中发现scrapy中使用，logger.cfg的文件的位置还有要求，因为scrapy和普通的python脚本根目录是有区别的 下图绿色字体表示scrapy的根是与scrapy.cfg的直接上级目录 而红色字体的根是与settings.py的直接上级目录， 这里是有区别的，要想通用，最简单的办法，就是在这两个根目录下面都加上logger.cfg文
scrapy的日志logging的使用
huangwencai123的博客
04-27 1429
scrapy的日志logging的使用 1、 意义 ： 为了方便在调试。在scrapy运行中可能停不下来， 关闭就没有记录了。因此把日志保存在文本中方便看信息。 2 、使用： 1） 在settings中增加： # log config import datetime Today=datetime.datetime.now()#取得现在的时间 Log_file_path='scrapy_{}_{}_...
基于Python的Scrapy爬虫框架设计源码
04-18
本项目是基于Python的Scrapy爬虫框架设计源码，包含22个文件，其中主要包含12个py源代码文件，4个xml配置文件等。系统采用了Python编程语言，实现了网站爬虫的功能，可以高效地抓取网站数据。项目结构清晰，代码...
完整版Python网络爬虫之Scrapy爬虫框架使用案例教程含源代码共18页.pdf
06-19
Scratch，是抓取的意思，这个Python的爬虫框架叫Scrapy，大概也是这个意思吧，就叫它：小刮刮吧。 小刮刮是一个为遍历爬行网站、分解获取数据而设计的应用程序框架，它可以应用在广泛领域：数据挖掘、信息处理和或者...
Python笔记——scrapy爬虫框架
01-06
1. scrapy框架介绍 — — python使用的最广泛的爬虫框架。 2. 创建项目：终端cmd下创建 输入命令：scrapy startproject [项目名qsbk] 生成目录结构： 1、scrapy.cfg:项目配置文件 2、items.py :定义需要爬去的字段 3...
Python编程：scrapy中使用logging，日志出现多次
彭世瑜的博客
04-09 1527
同时使用scrapy.log和logging，日志会多次出现在控制台 通过 logger 成功解决 import logging logger = logging.getLogger(__name__) logger.setLevel(level=logging.DEBUG) logger.debug("message") ...
python log日志系列之scrapy
suelai的博客
03-06 1240
log 信息级别从低到高依次是：notset、debug、info、warn、error、critical，日志只会显示比设置的日志等级高的级别，等级低的不会显示。scrapy中：    默认终端显示的是debug级别的log信息    在setting里面设置     LOG_LEVEL = WARN #log日志的级别    LOG_FILE = "./..."  # log日志的存储位置在s...
Scrapy 中 logging日志的应用
weixin_41245276的博客
02-18 182
Scrapy提供5层logging级别: CRITICAL - 严重错误(critical) ERROR - 一般错误(regular errors) WARNING - 警告信息(warning messages) INFO - 一般信息(informational messages) DEBUG - 调试信息(debugging messages) ...
scrapy框架—logging模块的使用
沉梦昂志
07-10 184
1. logging模块的使用 普通代码中使用日志： 其他代码调用
Scrapy 中 log 功能
Great Expectations的博客
08-22 1084
Scrapy提供了log功能，可以通过 logging 模块使用 可以在settings 中添加使用 LOG_FILE = "mySpider.log" LOG_LEVEL = "INFO" LOG_FILE 储存log文件路径 LOG_LEVEL 日志等级 Scrapy提供5层logging级别: CRITICAL - 严重错误(critical) ERROR - 一般错误...
Scrapy之日志文件的产生
lyx的专栏
06-23 9528
我们写爬虫的时候，也许会需要记录一些数据或者异常信息，通常这些数据我们会希望以日志的形式保存下来。 首先，我们在需要产生日志的python文件开头引入一个logging包。 import logging然后，我们可以用 self.log("your log information")这样的方式产生你的log文件。 当然，我们还需要一些设置，就是修改一些我们的setting文件。我们需要在s
Scrapy笔记一：Scrapy的介绍
xiaoduu的博客
07-12 83
文章目录Scrapy的介绍为什么要学Scrapy什么是Scrapy框架Twisted异步网络框架同异步和非阻塞同步和异步同步异步阻塞和非阻塞阻塞非阻塞 Scrapy的介绍 为什么要学Scrapy 让爬虫更快更强 什么是Scrapy 框架 Twisted异步网络框架 同异步和非阻塞 同步和异步 同步 发送，必须要有返回值 异步 发送，有没有返回都可以 阻塞和非阻塞 阻塞 阻塞类似：input 非阻塞 非阻塞类似：print ...
python 日志logging模块的使用
rosefun96的博客
01-29 1549
1、简介 import logging from scrapy.utils.log import configure_logging configure_logging(install_root_handler=False) logging.basicConfig( filename='log.txt', format='%(levelname)s: %(message)s
scrapy框架常用从头到尾巴
weixin_45410801的博客
09-05 541
总结scrapy框架 一. 核心组件和中间件 引擎 engine 自动运行，无需关祖，会自动组织所有的请求对象，分发给下载器 下载器 downloader 从引擎处获取到请求对象后，请求数据 爬虫 spiders scrapy.Spider 普通的爬虫 scrapy.CrawSpider 可设置规则的爬虫类 Rule规则类 start_requests()开始函数 调度器 scheduler 管道 Item pipeline 清理HTML数据 验证爬去的数据(检查item包含某些字段)
Python Scrapy爬虫框架入门教程
"这是一份关于Python爬虫入门的内训教程，重点介绍了Scrapy爬虫框架，并补充了Python基础知识，包括列表、元组、字符串的切片操作以及列表生成式和生成器的概念。此外，还提及了协程在Python中的应用，作为异步编程...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值