scrapy-from_crawler实例化Spider

最新推荐文章于 2024-07-07 20:53:23 发布

moisiet

最新推荐文章于 2024-07-07 20:53:23 发布

阅读量2.2k

点赞数 1

分类专栏：爬虫文章标签： scarpy from_crawler

本文链接：https://blog.csdn.net/qq_27056805/article/details/86262685

版权

本文探讨了Scrapy框架中Spider的初始化过程，重点解析了`from_crawler()`、`_set_crawler()`和`init_()`三个关键函数的作用和流程，帮助读者深入理解Scrapy的内部工作机制。

摘要由CSDN通过智能技术生成

spider的初始化是在Crawler类内完成的。而且是调用Spider内的类方法from_crawler()初始化spider的，下面分析这个过程的流程。Spider源码如下：

class Spider(object_ref):
    """Base class for scrapy spiders. All spiders must inherit from this
    class.
    """

    name = None
    custom_settings = None

    def __init__(self, name=None, **kwargs):
        if name is not None:
            self.name = name
        elif not getattr(self, 'name', None):
            raise ValueError("%s must have a name" % type(self).__name__)
        self.__dict__.update(kwargs)
        if not hasattr(self, 'start_urls'):
            self.start_urls = []

    @property
    def logger(self):
        logger = logging.getLogger(self.name)
        return logging.LoggerAdapter(logger, {'spider': self})

    def log(self, message, level=logging.DEBUG, **kw):
        """Log the given message at the given log level

        This helpe