1: pip install wheel
2: pip install lxml
3: pip install PyOpenssl
4. 安装scrapy模块需要安装安装安装 microsoft visual c++ build tools(直接安装个Visual Studio)
即可。
5.安装twisted模块(pip install twisted),可能出错
Running setup.py bdist_wheel for Twisted … error
Failed building wheel for Twisted
Running setup.py clean for Twisted
Failed to build Twisted
Installing collected packages: pycparser, cffi, asn1crypto, cryptography, pyOpenSSL, Twisted, pyasn1, pyasn1-modules, service-identity, queuelib, scrapy
Running setup.py install for Twisted … error....
修改该文件第75行的“UTF-8”为”mbcs", 安装成功scrapy,改回UTF-8.
6 .安装twisted模块成功之后,直接安装scrapy模块(pip install scapy).
测试是否安装成功
在命令行中,首先创建项目
scrapy startproject hello
进入项目目录
cd hello
生成爬虫
scrapy genspider baidu www.baidu.com
爬取百度
scrapy crawl baidu
如果命令都可以正常运行,则证明安装成功。
参考:https://blog.csdn.net/qq_29186489/article/details/78736945