python scrapy菜鸟教程_python学习-scrapy学习笔记

python-scrapy学习笔记

一、你可以为你的spider指定处理数据的pipeline，不过这需要一些代码

首先我们需要一个装饰器(decorator)，这个装饰器放到pipeline文件中，类的外部，因为多个pipeline需要用到这个装饰器

def check_spider_pipeline(process_item_method):

"""该注解用在pipeline上

:param process_item_method:

:return:

"""

@functools.wraps(process_item_method)

def wrapper(self, item, spider):

# message template for debugging

msg = "{0} pipeline step".format(self.__class__.__name__)

# if class is in the spider"s pipeline, then use the

# process_item method normally.

if self.__class__ in spider.pipeline:

logging.info(msg.format("executing"))

return process_item_method(self, item, spider)

# otherwise, just return the untouched item (skip this step in

# the pipeline)

else:

logging.info(msg