一、现象:Scrapy运行时提示 No module named _sqlite3,但是在python中import sqlite3没有问题。
2016-09-27 09:34:14 [scrapy] INFO: Scrapy 1.1.3 started (bot: IBDP)
2016-09-27 09:34:14 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'IBDP.spiders', 'LOG_LEVEL': 'INFO', 'CO
NCURRENT_REQUESTS': 1, 'SPIDER_MODULES': ['IBDP.spiders'], 'BOT_NAME': 'IBDP', 'DOWNLOAD_DELAY': 10}
2016-09-27 09:34:14 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2016-09-27 09:34:14 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-09-27 09:34:14 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2016-09-27 09:34:14 [scrapy] INFO: Enabled item pipelines:
['IBDP.pipelines.MySQLPipeline']
2016-09-27 09:34:14 [scrapy] INFO: Spider opened
Unhandled error in Deferred:
2016-09-27 09:34:14 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "/usr/local/python2.7/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 57, in run
self.crawler_process.crawl(spname, **opts.spargs)
File "/usr/local/python2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 163, in crawl
return self._crawl(crawler, *args, **kwargs)
File "/usr/local/python2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 167, in _crawl
d = crawler.crawl(*args, **kwargs)
File "/usr/local/python2.7/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/defer
.py", line 1274, in unwindGenerator
return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
File "/usr/local/python2.7/lib/python2.7/site-packages/Twisted-15.2.1-py2.7-linux-x86_64.egg/twisted/internet/defer
.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/usr/local/python2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 90, in crawl
six.reraise(*exc_info)
File "/usr/local/python2.7/lib/python2.7/site-packages/scrapy/crawler.py", line 74, in crawl
yield self.engine.open_spider(self.spider, start_requests)
exceptions.ImportError: No module named _sqlite3
2016-09-27 09:34:14 [twisted] CRITICAL:
二、解决方法:
安装 sqlite-devel,重新安装python
yum install sqlite-devel
cd Python-2.7.11
make
make install