Scrapy 1.2 + Python 3.5 在 Windows 上的安装方法

Python 同时被 3 个专栏收录
4 篇文章 0 订阅
3 篇文章 0 订阅
3 篇文章 0 订阅

1.安装 Python 3.5 for windows,详细的安装方法可以参见关联Url

2.打开cmd 输入 pip3 install scrapy -i https://pypi.douban.com/simple,开始安装进程(这里使用豆瓣的源,速度比较快)

耐心等待下载安装完成,如果下载失败,可以多试几次

注:

如果出现

You are using pip version x.x.x, however version x.x.x is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

这个警告的话,可以执行一次 python -m pip install --upgrade pip -i https://pypi.douban.com/simple 将 pip更新到最新版本,就没有该警告了

如果出现

error: Unable to find vcvarsall.bat这个错误,请参见 https://blogs.msdn.microsoft.com/pythonengineering/2016/04/11/unable-to-find-vcvarsall-bat/#comments

如果出现

fatal error C1083: 无法打开包括文件: “libxml/xxx.h”: No such file or directory这个错误,证明你的libxml没有编译,可以从http://download.csdn.net/detail/zs808/9545177 该处下载编译好的libxml库。然后将libs目录下的文件拷贝到你的VC编译器目录下的libs目录中,将include下的文件拷贝到VC编译器下的includes目录中,重新安装即可。

3.安装好后输入 scrapy bench 执行scrapy基准测试,如果出现以下结果:


2016-06-09 13:58:33 [scrapy] INFO: Scrapy 1.1.0 started (bot: scrapybot)
2016-06-09 13:58:36 [scrapy] INFO: Overridden settings: {'LOG_LEVEL': 'INFO', 'LOGSTATS_INTERVAL': 1, 'CLOSESPIDER_TIMEOUT': 10}
2016-06-09 13:58:38 [scrapy] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
2016-06-09 13:58:39 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2016-06-09 13:58:39 [scrapy] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']                                                                     

2016-06-09 13:58:39 [scrapy] INFO: Enabled item pipelines:
[]
2016-06-09 13:58:39 [scrapy] INFO: Spider opened
2016-06-09 13:58:39 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:40 [scrapy] INFO: Crawled 50 pages (at 3000 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:41 [scrapy] INFO: Crawled 90 pages (at 2400 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:42 [scrapy] INFO: Crawled 122 pages (at 1920 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:43 [scrapy] INFO: Crawled 157 pages (at 2100 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:44 [scrapy] INFO: Crawled 186 pages (at 1740 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:45 [scrapy] INFO: Crawled 221 pages (at 2100 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:46 [scrapy] INFO: Crawled 250 pages (at 1740 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:47 [scrapy] INFO: Crawled 277 pages (at 1620 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:48 [scrapy] INFO: Crawled 306 pages (at 1740 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:49 [scrapy] INFO: Closing spider (closespider_timeout)
2016-06-09 13:58:49 [scrapy] INFO: Crawled 333 pages (at 1620 pages/min), scraped 0 items (at 0 items/min)
2016-06-09 13:58:50 [scrapy] INFO: Crawled 349 pages (at 960 pages/min), scraped 0 items (at 0 items/min)

2016-06-09 13:58:50 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 139704,
 'downloader/request_count': 349,
 'downloader/request_method_count/GET': 349,
 'downloader/response_bytes': 917753,
 'downloader/response_count': 349,
 'downloader/response_status_count/200': 349,
 'finish_reason': 'closespider_timeout',
 'finish_time': datetime.datetime(2016, 6, 9, 5, 58, 50, 287394),
 'log_count/INFO': 18,
 'request_depth_max': 15,
 'response_received_count': 349,
 'scheduler/dequeued': 349,
 'scheduler/dequeued/memory': 349,
 'scheduler/enqueued': 6979,
 'scheduler/enqueued/memory': 6979,
 'start_time': datetime.datetime(2016, 6, 9, 5, 58, 39, 212930)}
2016-06-09 13:58:50 [scrapy] INFO: Spider closed (closespider_timeout)

说明Scrapy已经成功安装,尽情使用吧。

注:如果Scrapy基准测试出现:ImportError: Can not import name '_win32studio'错误,则还需要执行pip install twisted-win -i https://pypi.douban.com/simple


  • 0
    点赞
  • 2
    评论
  • 0
    收藏
  • 打赏
    打赏
  • 扫一扫,分享海报

评论 2 您还未登录,请先 登录 后发表或查看评论
©️2022 CSDN 皮肤主题:大白 设计师:CSDN官方博客 返回首页

打赏作者

zs808

你的鼓励将是我创作的最大动力

¥2 ¥4 ¥6 ¥10 ¥20
输入1-500的整数
余额支付 (余额:-- )
扫码支付
扫码支付:¥2
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值