I am having some trouble with a scrapy pipeline. My information is being scraped form sites ok and the process_item method is being called correctly. However the spider_opened and spider_closed methods are not being called.
class MyPipeline(object):
def __init__(self):
log.msg("Initializing Pipeline")
self.conn = None
self.cur = None
def spider_opened(self, spider):
log.msg("Pipeline.spider_opened called", level=log.DEBUG)
def spider_closed(self, spider):
log.msg("Pipeline.spider_closed called", level=log.DEBUG)
def process_item(self, item, spider):
log.msg("Processsing item " + item['title'], level=log.DEBUG)
Both the __init__
and process_item
logging messages are displyed in the log, but the spider_open
and spider_close
logging messages are not.
I need to use the spider_opened and spider_closed methods as I want to use them to open and close a connection to a database, but nothing is showing up in the log for them.
If anyone has any suggested that would be very useful.
dispatcher
variable? And how come I can't find this indoc.scrapy.org/en/latest/topics/item-pipeline.html? :( – wrongusername Oct 8 '12 at 18:05from scrapy.xlib.pydispatch import dispatcher
from scrapy import signals
– herrherr Oct 28 '13 at 15:08