scrapy [boto] ERROR: Caught exception reading instance data URLError: <urlopen error [Errno 10051] >

最新推荐文章于 2023-03-30 19:10:56 发布

一字腾云

最新推荐文章于 2023-03-30 19:10:56 发布

阅读量3.3k

点赞数 1

分类专栏：爬虫文章标签： python scrapy

本文链接：https://blog.csdn.net/liyuetao680/article/details/48313313

版权

爬虫专栏收录该内容

16 篇文章 0 订阅

订阅专栏

执行过程中出现错误：

2015-09-09 11:13:26 [boto] DEBUG: Retrieving credentials from metadata server.
2015-09-09 11:13:27 [boto] ERROR: Caught exception reading instance data
Traceback (most recent call last):
  File "D:\anzhuang\Anaconda\lib\site-packages\boto\utils.py", line 210, in retry_url
    r = opener.open(req, timeout=timeout)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 431, in open
    response = self._open(req, data)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 449, in _open
    '_open', req)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 1227, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "D:\anzhuang\Anaconda\lib\urllib2.py", line 1197, in do_open
    raise URLError(err)
URLError: <urlopen error timed out>
2015-09-09 11:13:27 [boto] ERROR: Unable to read instance data, giving up

在setting.py中禁用s3 download就可以了
解决方法：

DOWNLOAD_HANDLERS = {'S3': None,}

在stackoverflow上有些解释,有时间可以看看这篇错误分析错误
下面是我贴的部分内容。
That particular error message is being generated by boto (boto 2.38.0 py27_0), which is used to connect to Amazon S3. Scrapy doesn’t have this enabled by default.

EDIT: In reply to the comments, this appears to be a bug with Scrapy when boto is present (bug here).

In response “how to disable the Download handler”, add the following to your settings.py file:

DOWNLOAD_HANDLERS = {
        'S3':None,
}

Your settings.py file should be in the root of your Scrapy project folder, (one level deeper than your scrapy.cfg file).

If you’ve already got DOWNLOAD_HANDLERS in your settings.py file, just add a new entry for ‘s3’ with a None value.

EDIT 2: I’d highly recommend looking at setting up virtual environments for your projects. Look into virtualenv, and it’s usage. I’d make this recommendation regardless of packages used for this project, but doubly so with your extreme number of packages.