小书死磕爬虫,学会一样,再攻一样。今天学习scrapy 时安装就出错了,不能忍,开搞pip install scrapy Collecting scrapy Downloading https://files.pythonhosted.org/packages/5d/12/a6197eaf97385e96fd8ec56627749a6229a9b3178ad73866a0b1fb377379/Scrapy-1.5.1-py2.py3-none-any.whl (249kB) 100% |████████████████████████████████| 256kB 410kB/s Collecting parsel>=1.1 (from scrapy) Downloading https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl .................................... 出错啦!!!!!
解决问题吧:开搞啦
第一步: 先找到scrapy的官网如:(https://doc.scrapy.org/en/master/intro/install.html#installing-scrapy)
第二步:往下滑你会看到:
也就是说这里面涉及到的东西你都需要安装,这个很好用,你直接点pyOpenSSL就会跳到另一个网址,里面有安装代码,直接复制即可。
如果你都能成功安装上那就没问题啦!
小书还要第三步呢!!!
安装过程中:Twisted报错,效果如下:
Could not find a version that satisfies the requirement Twisted-18.9.0-cp36-cp36m-win_amd64 (from versions: )
No matching distribution found for Twisted-18.9.0-cp36-cp36m-win_amd64
解决吧:
我找到Twisted的网址(https://pypi.org/project/Twisted/)
安装很顺利的出错啦!!!
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools
----------------------------------------
Command "c:\users\shu\appdata\local\programs\python\python36\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\Shu\\AppData\\Local\\Temp\\pip-install-w62o51mg\\Twisted\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\Shu\AppData\Local\Temp\pip-record-tj7glxk8\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\Shu\AppData\Local\Temp\pip-install-w62o51mg\Twisted\
奔溃>>>>>>>>>>>>>此处已过去两小时
第三步:安装Twisted
先解决一个问题,这儿说什么缺少一个 Microsoft Visual C++ 14.0环境,听我的,别去搞,我花了一个积分下载了别人的一个,包都是错的。我们需要做的是进入下面这个官网:[(https://www.lfd.uci.edu/~gohlke/pythonlibs/)
去里面找到,找到与你电脑版本和python版本一致的whl文件下载
下载好后,存放在电脑桌面,然后开始安装:
C:\Users\Shu\Desktop>pip install C:\Users\Shu\Desktop\Twisted-18.9.0-cp36-cp36m-win_amd64.whl
Processing c:\users\shu\desktop\twisted-18.9.0-cp36-cp36m-win_amd64.whl
Requirement already satisfied: Automat>=0.3.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (0.7.0)
Requirement already satisfied: constantly>=15.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (15.1.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (1.9.0)
Requirement already satisfied: attrs>=17.4.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (18.2.0)
Requirement already satisfied: incremental>=16.10.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (17.5.0)
Requirement already satisfied: hyperlink>=17.1.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (18.0.0)
Requirement already satisfied: zope.interface>=4.4.2 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted==18.9.0) (4.6.0)
Requirement already satisfied: six in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Automat>=0.3.0->Twisted==18.9.0) (1.11.0)
Requirement already satisfied: setuptools in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from PyHamcrest>=1.9.0->Twisted==18.9.0) (39.1.0)
Requirement already satisfied: idna>=2.5 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from hyperlink>=17.1.1->Twisted==18.9.0) (2.7)
Installing collected packages: Twisted
Successfully installed Twisted-18.9.0
成功啦!!!!!!开不开心?
第四步:安装Scrapy
C:\Users\Shu\Desktop>pip install scrapy
Collecting scrapy
Using cached https://files.pythonhosted.org/packages/5d/12/a6197eaf97385e96fd8ec56627749a6229a9b3178ad73866a0b1fb377379/Scrapy-1.5.1-py2.py3-none-any.whl
Requirement already satisfied: six>=1.5.2 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (1.11.0)
Requirement already satisfied: cssselect>=0.9 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (1.0.3)
Collecting queuelib (from scrapy)
Using cached https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
Using cached https://files.pythonhosted.org/packages/29/fa/995e364220979e577e7ca232440961db0bf996b6edaf586a7d1bd14d81f1/service_identity-17.0.0-py2.py3-none-any.whl
Requirement already satisfied: parsel>=1.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (1.5.1)
Requirement already satisfied: w3lib>=1.17.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (1.19.0)
Collecting PyDispatcher>=2.0.5 (from scrapy)
Requirement already satisfied: Twisted>=13.1.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (18.9.0)
Requirement already satisfied: lxml in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (4.2.5)
Requirement already satisfied: pyOpenSSL in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from scrapy) (18.0.0)
Collecting pyasn1-modules (from service-identity->scrapy)
Using cached https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl
Collecting pyasn1 (from service-identity->scrapy)
Using cached https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl
Requirement already satisfied: attrs in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from service-identity->scrapy) (18.2.0)
Requirement already satisfied: incremental>=16.10.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (17.5.0)
Requirement already satisfied: constantly>=15.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (15.1.0)
Requirement already satisfied: zope.interface>=4.4.2 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (4.6.0)
Requirement already satisfied: PyHamcrest>=1.9.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (1.9.0)
Requirement already satisfied: Automat>=0.3.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (0.7.0)
Requirement already satisfied: hyperlink>=17.1.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from Twisted>=13.1.0->scrapy) (18.0.0)
Requirement already satisfied: cryptography>=2.2.1 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from pyOpenSSL->scrapy) (2.3.1)
Requirement already satisfied: setuptools in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->scrapy) (39.1.0)
Requirement already satisfied: idna>=2.5 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from hyperlink>=17.1.1->Twisted>=13.1.0->scrapy) (2.7)
Requirement already satisfied: asn1crypto>=0.21.0 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from cryptography>=2.2.1->pyOpenSSL->scrapy) (0.24.0)
Requirement already satisfied: cffi!=1.11.3,>=1.7 in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from cryptography>=2.2.1->pyOpenSSL->scrapy) (1.11.5)
Requirement already satisfied: pycparser in c:\users\shu\appdata\local\programs\python\python36\lib\site-packages (from cffi!=1.11.3,>=1.7->cryptography>=2.2.1->pyOpenSSL->scrapy) (2.19)
Installing collected packages: queuelib, pyasn1, pyasn1-modules, service-identity, PyDispatcher, scrapy
Successfully installed PyDispatcher-2.0.5 pyasn1-0.4.4 pyasn1-modules-0.2.2 queuelib-1.5.0 scrapy-1.5.1 service-identity-17.0.0
成功安装!!!!
第五步:测试
C:\Users\Shu\Desktop>python
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>>
好开森!!!
希望我的经历对你有所帮助,小书,一个21世纪的python小白,我坚信知行合一,知道并且做到,让我们一起努力,为美好的未来奋斗吧!!!(这个插入代码部分今天刚学会,不知道效果如何)