pyhton爬虫学习（十一）：Scrapy的安装

我现在的python版本为3.7.3 可直接在命令行输入pip install scrapy进行scrapy的安装，安装的时候会自动检索和安装所需要的其他依赖包。不过正常安装也可能会出现一些错误导致安装失败。比如我在window安装的时候提示提示如下错误

Collecting Twisted>=13.1.0 (from scrapy)   Downloading https://files.pythonhosted.org/packages/f8/2b/a80a70f71eb2b86992ffa5aaae41457791ae67faa70927fd16b76127c2b7/Twisted-19.2.0.tar.bz2 (3.1MB)     100% |████████████████████████████████| 3.1MB 455kB/s     
Complete output from command python setup.py egginfo:    
Traceback (most recent call last):     
File “<string>”, line 1, in <module>     
File “C:\Users\z\AppData\Local\Temp\pip-install-zd6ypw3\Twisted\setup.py”, line 20, in <module>         setuptools.setup(**setup“getSetupArgs”)      
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools_init.py”, line 144, in setup        
_install_setup_requires(attrs)       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\init.py”, line 139, in _install_setup_requires         
dist.fetch_build_eggs(dist.setup_requires)       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\dist.py”, line 724, in fetch_build_eggs         
replace_conflicting=True,       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\pkg_resources\init.py”, line 782, in resolve         
replace_conflicting=replace_conflicting       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\pkg_resources\init.py”, line 1065, in best_match        
return self.obtain(req, installer)       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\pkg_resources__init.py”, line 1077, in obtain        
return installer(requirement)      
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\dist.py”, line 791, in fetch_build_egg         
return cmd.easy_install(req)       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\command\easy_install.py”, line 667, in easy_install         not self.always_copy, self.local_index       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\package_index.py”, line 654, in fetch_distribution         self.find_packages(requirement)       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\package_index.py”, line 487, in find_packages         self.scan_url(self.index_url + requirement.unsafe_name + ‘/‘)      
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\package_index.py”, line 827, in scan_url        
self.process_url(url, True)       
File “c:\users\z\appdata\local\programs\python\python37\lib\site-packages\setuptools\package_index.py”, line 357, in process_url         
page = f.read()       
File “c:\users\z\appdata\local\programs\python\python37\lib\http\client.py”, line 460, in read         
s = self._safe_read(self.length)       
File “c:\users\z\appdata\local\programs\python\python37\lib\http\client.py”, line 610, in _safe_read         chunk = self.fp.read(min(amt, MAXAMOUNT))      
File “c:\users\z\appdata\local\programs\python\python37\lib\socket.py”, line 589, in readinto         return self._sock.recv_into(b)       
File “c:\users\z\appdata\local\programs\python\python37\lib\ssl.py”, line 1052, in recv_into         return self.read(nbytes, buffer)       
File “c:\users\z\appdata\local\programs\python\python37\lib\ssl.py”, line 911, in read         
return self._sslobj.read(len, buffer)     
socket.timeout: The read operation timed out    
 ————————————————————
  Command “python setup.py egg_info” failed with error code 1 in C:\Users\xxx\AppData\Local\Temp\pip-install-zd6yp_w3\Twisted\

注意到这里显示一个timed out，就是超时的意思，一般是网络的原因，这个时候可以重新输入pip install scrapy重试一次，已经安装过的包不会再重新安装。所以不用担心会重复安装需要的依赖包。 当然，如果是以前的版本，或者网络的原因，可能pip install scrapy重试多次都失败，那怎么办呢？这里可以选择本地安装。就是先到其他网站下载安装失败的依赖包到本地，在进行本地安装即可。 这个可以给一个可以下载相关依赖包的一个辅助网站的地址： https://www.lfd.uci.edu/~gohlke/pythonlibs/

比如，我确实lxml的依赖包。可以到这个网上上面找到lxml的位置，点击进去，如下图

因为我是python3.7.3,winown64位的系统，所以这里我选择lxml 4.3.3 cp37 cp37m win_amd64.whl（为什么选择这个应该很容易看出来吧）。

下载到本地某个文件，通过cd命令进入该文件所在目录，然后
pip install lxml-4.3.3-cp37-cp37m-win_amd64.whl
即可实现本地安装。 安装scrapy完成之后，显示如下。主要展示了安装的所有的依赖包

这个时候输入scrapy -V，显示如下，说scrapy已经安装成功。

Mac安装scrapy的时候也是用pip install scrapy可以轻松安装成功。不过这个建议还是都安装当前最新的python3.7.3版本，旧版本经常会出现某些依赖包安装失败的情况，比较麻烦。

欢迎关注本人公众号和小程序，谢谢