2021-01-29

于 2021-01-29 16:00:18 发布
阅读量775 收藏
点赞数 2
文章标签： 其他 python redis
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/ghostvalleyqq/article/details/113392177
版权

NameError: Module 'scrapy_redis.dupefilter' doesn't define any object named 'RFPDuperFilter'

运行RedisCrawlSpider时报错如下：

(crawlvenv) [hdfs@miaoshou1x spiders]$ scrapy runspider zce_spider1.py 
2021-01-29 15:30:38 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: futures_market)
2021-01-29 15:30:38 [scrapy.utils.log] INFO: Versions: lxml 4.6.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.0 (default, Jan 28 2021, 23:24:42) - [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1i  8 Dec 2020), cryptography 3.3.1, Platform Linux-3.10.0-1062.el7.x86_64-x86_64-with-centos-7.7.1908-Core
2021-01-29 15:30:38 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2021-01-29 15:30:38 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'futures_market',
 'DOWNLOAD_DELAY': 2,
 'DUPEFILTER_CLASS': 'scrapy_redis.dupefilter.RFPDuperFilter',
 'NEWSPIDER_MODULE': 'futures_market.spiders',
 'SCHEDULER': 'scrapy_redis.scheduler.Scheduler',
 'SPIDER_LOADER_WARN_ONLY': True,
 'SPIDER_MODULES': ['futures_market.spiders']}
2021-01-29 15:30:38 [scrapy.extensions.telnet] INFO: Telnet Password: a0e84c030ce0acdf
2021-01-29 15:30:38 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2021-01-29 15:30:38 [zce_spider1] INFO: Reading start URLs from redis key 'zce:start_urls' (batch size: 16, encoding: utf-8
2021-01-29 15:30:38 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:57145/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions": {"extensions": [], "args": ["--headless", "--no-sandbox", "--disable-dev-shm-usage", "blink-settings=imagesEnabled=false", "--disable-gpu"]}}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions": {"extensions": [], "args": ["--headless", "--no-sandbox", "--disable-dev-shm-usage", "blink-settings=imagesEnabled=false", "--disable-gpu"]}}}
2021-01-29 15:30:38 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 127.0.0.1:57145
2021-01-29 15:30:38 [urllib3.connectionpool] DEBUG: http://127.0.0.1:57145 "POST /session HTTP/1.1" 200 716
2021-01-29 15:30:38 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2021-01-29 15:30:38 [scrapy.middleware] INFO: Enabled downloader middlewares:
['futures_market.middlewares.ZCESpiderSeleniumDownloaderMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2021-01-29 15:30:38 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2021-01-29 15:30:38 [scrapy.middleware] INFO: Enabled item pipelines:
['scrapy_redis.pipelines.RedisPipeline']
2021-01-29 15:30:38 [scrapy.core.engine] INFO: Spider opened
2021-01-29 15:30:38 [scrapy.core.engine] INFO: Closing spider (shutdown)
2021-01-29 15:30:38 [scrapy.core.engine] ERROR: Scraper close failure
Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/utils/misc.py", line 65, in load_object
    obj = getattr(mod, name)
AttributeError: module 'scrapy_redis.dupefilter' has no attribute 'RFPDuperFilter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
NameError: Module 'scrapy_redis.dupefilter' doesn't define any object named 'RFPDuperFilter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/core/engine.py", line 325, in <lambda>
    dfd.addBoth(lambda _: self.scraper.close_spider(spider))
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/core/scraper.py", line 86, in close_spider
    slot.closing = defer.Deferred()
AttributeError: 'NoneType' object has no attribute 'closing'
2021-01-29 15:30:38 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method CoreStats.spider_closed of <scrapy.extensions.corestats.CoreStats object at 0x7fb346a40860>>
Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/utils/misc.py", line 65, in load_object
    obj = getattr(mod, name)
AttributeError: module 'scrapy_redis.dupefilter' has no attribute 'RFPDuperFilter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
NameError: Module 'scrapy_redis.dupefilter' doesn't define any object named 'RFPDuperFilter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/utils/defer.py", line 157, in maybeDeferred_coro
    result = f(*args, **kw)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/extensions/corestats.py", line 31, in spider_closed
    elapsed_time = finish_time - self.start_time
TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
2021-01-29 15:30:38 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'log_count/DEBUG': 4, 'log_count/ERROR': 2, 'log_count/INFO': 9}
2021-01-29 15:30:38 [scrapy.core.engine] INFO: Spider closed (shutdown)
Unhandled error in Deferred:
2021-01-29 15:30:38 [twisted] CRITICAL: Unhandled error in Deferred:

Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
    _inlineCallbacks(None, g, status)
--- <exception caught here> ---
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
builtins.NameError: Module 'scrapy_redis.dupefilter' doesn't define any object named 'RFPDuperFilter'

2021-01-29 15:30:38 [twisted] CRITICAL: 
Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/utils/misc.py", line 65, in load_object
    obj = getattr(mod, name)
AttributeError: module 'scrapy_redis.dupefilter' has no attribute 'RFPDuperFilter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/home/hdfs/.virtualenvs/crawlvenv/lib/python3.7/site-packages/scrapy/crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
NameError: Module 'scrapy_redis.dupefilter' doesn't define any object named 'RFPDuperFilter'

根本没有提及具体是哪个模块错了，看着这些报错的路径都是解释器的问题，可能是想说我的程序在解释器里的底层代码跑不通吧。但是

scrapy_redis.dupefilter.RFPDuperFilter

我看着眼熟。找来找去。。

原因：

# Scrapy-Redis相关配置
# 调度器配置，将之前默认的scrapy调度器改为Scrapy-Redis组件提供的调度器，确保request存储到redis中
SCHEDULER = "scrapy_redis.scheduler.Scheduler"

# 使用Scrapy-Redis组件提供的去重类型
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDuperFilter"

# 将item pipeline 设置为 Scrapy-Redis组件RedisPipeline
ITEM_PIPELINES = {
   # 'fang.pipelines.FangPipeline': 300,
   'scrapy_redis.pipelines.RedisPipeline': 300,
}

# 调度缓存设置，用于在Redis中保持Scrapy-Redis组件用到的队列，
# 缓存队列中的数据，从而可以实现暂停和恢复的功能。（以免总是从第一个url开始重复爬取数据）
SCHEDULER_PERSIST = True

# 设置Redis的连接信息
REDIS_HOST = '192.168.1.21'
REDIS_PORT = 6379

在settings文件里这段专门给scrapy_redis模块复制粘贴的代码里出问题了！

DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDuperFilter"

RFPDuperFilter，字母拼写错误，改成RFPDupeFilter如下：

DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"

最终运行成功了，真的一点都不能错呀，以后碰到这种情况要把报错的地方仔细看，看是否有空格就或拼写错误。把报错的内容到各个模块下查找。

ghostvalleyqq
关注
  • 2
    点赞
  • 踩
  • 0
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
【求助】 python3-scrapy [twisted] CRITICAL: Unhandled error in Deferred
Darkofnoon的博客
04-29 1233
我安装pycharm后用内部terminal可以运行，应该是window shell的问题，希望有懂的大佬分享一下原因，谢谢 2019.4.30 -----------------------------------------------------------------------------------------------------------------------------...
Scrapy爬取数据[scrapy.core.scraper] ERROR: Spider error processing解决办法
zp17834994071的博客
08-26 1万+
在爬取数据的时候遇到会遇到很多各种各样的问题，但是有一个问题会经常出现，并且原因还很难发现，下面就给大家说下这个问题的原因，以及解决办法，开始运行会出现这样的一个错误： [scrapy.core.scraper] ERROR: Spider error processing 爬取数据代码： 出现的报错问题详细信息： 这时显示的信息是有空值None出现,导致的，其实也不难理解，大家可以想想，当我们爬取完网页信息的时候，当爬到最后一页的页面时，接下来就没有下一页的信息，如果我们没有进行判断的话，就会继续爬下
（未解决）scrapy框架运行出现的问题AttributeError: 'MemoryUsage' object has no attribute 'tasks'
metamoss
11-05 3612
2018-11-05 23:43:25 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: moo) 2018-11-05 23:43:25 [scrapy.utils.log] INFO: Versions: lxml 4.2.5.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.5.1, w3lib 1....
Scrapy [twisted] CRITICAL: Unhandled error in Deferred:错误
u013358302的专栏
01-08 1万+
最近刚开始使用爬虫框架scrapy，开始就遇到这么个问题，严重挫伤学习心情，终于耗费一天时间解决这个问题 解决方法： 在安装scrapy的时候下载了pywin32，但是忘记安装这个模块 进入python27目录下，手动安装，可以在命令行使用python python27\scripts\pywin32_postinstall.py -install 之后再运行爬虫程序就ok了
NameError: Module 'scrapy_redis.scheduler' doesn't define any object named ' Scheduler'
weixin_43256057的博客
05-27 2503
运行RedisCrawlSpider时报错如下： Traceback (most recent call last): File “/usr/local/lib/python3.5/dist-packages/scrapy/crawler.py”, line 172, in crawl return self._crawl(crawler, *args, **kwargs) File “/usr/...
2021-01-29-BetterLab
05-25
木工车间模板 该存储库是The Carpentries（， 和的）模板，用于为研讨会创建网站。 请不要直接在GitHub上存储此存储库。 相反，请按照以下使用GitHub的“模板”功能来复制此workshop-template存储库并为您的工作坊...
Draft 2021-01-29 测试-数据集
03-18
本文将深入探讨“Draft 2021-01-29 测试-数据集”这一主题，特别是其中包含的“景物图片”——“云烟.png”和“花.jpg”。 首先，我们来看“数据集”这一概念。在机器学习和人工智能领域，数据集是指一组用于训练、...
gtk3-runtime-3.24.29-2021-04-29-ts-win64.exe
07-10
gtk3-runtime-3.24.29-2021-04-29-ts-win64.exe
minio-RELEASE.2023-01-31T02-24-19Z版本
02-08
新版本配置 export MINIO_ROOT_USER=admin export MINIO_ROOT_PASSWORD=12345678 原旧版本中的配置是MINIO_ACCESS_KEY和MINIO_SECRET_KEY，请注意别在各博客中直接复制把自己的配置写错了，新下载的minio是用不了旧...
2021-01-18-23-11-29-download-LeetDown_1.0.3.dmg
09-11
2021-01-18-23-11-29-download-LeetDown_1.0.3
Python获取Google sheet中的内容
简言
08-06 1万+
google 提供了各种API供开发者使用，之前介绍了map api的简单用法，这里介绍下googlesheet的api简单用法。
TypeError: method() missing 1 required positional argument: 'self'
C2496649361的博客
05-04 8690
学习python的时候把python学习手册上面的示例代码跟着敲了一遍。然后居然发现不能通过。检查之后发现确实不是抄错了，就是这行代码无法通过。我不知道作者的是如何通过的，反正我自己的编译器就是无法通过。首先上教材原代码： class Super: def method(self): print('in Super.method') def delegate...
ERROR: Exception: Traceback (most recent call last)的解决方法
热门推荐
inside802的博客
10-20 17万+
在使用pip安装pillow时，总是提示ERROR: Exception: Traceback (most recent call last) 到网上搜了很多方法，包括使用管理员权限，修改pip __main__方法，都不行，后来我想应该就是简单的网速低导致断链的问题吧？pip可能不支持断点续传。后来我按照cmd命令行里提示的下载地址，自己用浏览器下载，然后本地安装whl文件，成功。 下载完成...
3.27 udp聊天器出现Traceback (most recent call last):错误解决方案
巨基的博客
03-27 1万+
问题描述 跟udp聊天器 教学视频 ，同老师敲的同样的代码但出现了以下错误： Traceback (most recent call last): File “/home/python/PycharmProjects/untitled/05-udp；聊天器.py”, line 8, in dest_ip = input(“请输入对方的ip地址：”) File “”, line 1 192.168...
python爬虫之十一_scrapy框架经验、错误
IMchg的博客
11-05 4796
发是发发发的
java生成租金计划 需求：根据下面的条件生成每个月的月租金计划列表 递增周期：1年 租金递增率：6% 租赁开始时间 2021-03-01 租赁结束时间 2022-03-01 免租开始时间：2021-03-01 免租结束时间：2021-03-31 开始月租金：600 递增周期的时间是从租赁开始时间计算。
最新发布
05-26
2021-08-01 31 754.29 2021-09-01 30 796.39 2021-10-01 31 840.94 2021-11-01 30 887.02 2021-12-01 31 934.72 2022-01-01 31 984.12 2022-02-01 28 1035.30 ``` 可以看到，程序正确地根据递增周期和递增率...
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值