python 3.7下载安装scrapy_python3.7 Scrapy在Windows平台的安装

1.安装python并配置好环境变量

2.更新pip ，这步操作时遇到了问题，更新超时了，所以加上--default-timeout来解决超时问题：python -m pip --default-timeout=100 install --upgrade pip

3.安装lxml ：pip intsall lxml

4.安装wheel： pip install wheel

5.在python lib网站上以下安装包，然后使用pip install xxx 安装：

Twisted-18.9.0-cp37-cp37m-win_amd64.whl

pywin32-224-cp37-cp37m-win_amd64.whl

Scrapy-1.6.0-py2.py3-none-any.whl

Python Lib网站：https://www.lfd.uci.edu/~gohlke/pythonlibs/

6.Scrapy调试，验证是否安装成功：

C:\Users\admin>scrapy

Scrapy 1.6.0 - no active project

Usage:

scrapy [options] [args]

Available commands:

bench Run quick benchmark test

fetch Fetch a URL using the Scrapy downloader

genspider Generate new spider using pre-defined templates

runspider Run a self-contained spider (without creating a project)

settings Get settings values

shell Interactive scraping console

startproject Create new project

version Print Scrapy version

view Open URL in browser, as seen by Scrapy

[ more ] More commands available when run from project directory

Use "scrapy -h" to see more info about a command

部分安装日志如下：

C:\Users\admin>pip install pywin32-224-cp37-cp37m-win_amd64.whl

Processing c:\users\admin\pywin32-224-cp37-cp37m-win_amd64.whl

Installing collected packages: pywin32

Successfully installed pywin32-224

C:\Users\admin>pip install Scrapy-1.6.0-py2.py3-none-any.whl

Processing c:\users\admin\scrapy-1.6.0-py2.py3-none-any.whl

Collecting PyDispatcher>=2.0.5 (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz

Collecting pyOpenSSL (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)

100% |████████████████████████████████| 61kB 64kB/s

Collecting cssselect>=0.9 (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl

Collecting w3lib>=1.17.0 (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/81/43/9dcf92a77f5f0afe4f4df2407d7289eea01368a08b64bda00dd318ca62a6/w3lib-1.20.0-py2.py3-none-any.whl

Requirement already satisfied: Twisted>=13.1.0 in d:\python37\lib\site-packages (from Scrapy==1.6.0) (18.9.0)

Collecting service-identity (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/e9/7c/2195b890023e098f9618d43ebc337d83c8b38d414326685339eb024db2f6/service_identity-18.1.0-py2.py3-none-any.whl

Collecting queuelib (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl

Requirement already satisfied: six>=1.5.2 in d:\python37\lib\site-packages (from Scrapy==1.6.0) (1.12.0)

Requirement already satisfied: lxml in d:\python37\lib\site-packages (from Scrapy==1.6.0) (4.3.1)

Collecting parsel>=1.5 (from Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/96/69/d1d5dba5e4fecd41ffd71345863ed36a45975812c06ba77798fc15db6a64/parsel-1.5.1-py2.py3-none-any.whl

Collecting cryptography>=2.3 (from pyOpenSSL->Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/cd/ab/06deda5e26c4290d9973bbf2d53d09d09898f6f972576807c8e76ff13e76/cryptography-2.5-cp37-cp37m-win_amd64.whl (1.5MB)

100% |████████████████████████████████| 1.5MB 17kB/s

Requirement already satisfied: attrs>=17.4.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (18.2.0)

Requirement already satisfied: hyperlink>=17.1.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (18.0.0)

Requirement already satisfied: zope.interface>=4.4.2 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (4.6.0)

Requirement already satisfied: constantly>=15.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (15.1.0)

Requirement already satisfied: incremental>=16.10.1 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (17.5.0)

Requirement already satisfied: Automat>=0.3.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (0.7.0)

Requirement already satisfied: PyHamcrest>=1.9.0 in d:\python37\lib\site-packages (from Twisted>=13.1.0->Scrapy==1.6.0) (1.9.0)

Collecting pyasn1-modules (from service-identity->Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/da/98/8ddd9fa4d84065926832bcf2255a2b69f1d03330aa4d1c49cc7317ac888e/pyasn1_modules-0.2.4-py2.py3-none-any.whl (66kB)

100% |████████████████████████████████| 71kB 18kB/s

Collecting pyasn1 (from service-identity->Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/7b/7c/c9386b82a25115cccf1903441bba3cbadcfae7b678a20167347fa8ded34c/pyasn1-0.4.5-py2.py3-none-any.whl (73kB)

100% |████████████████████████████████| 81kB 13kB/s

Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/6b/a4/4d9ba87108405ef0612907c1da4595a2cb4fba91a18105e68c1b7751bfe3/cffi-1.12.0-cp37-cp37m-win_amd64.whl (170kB)

100% |████████████████████████████████| 174kB 11kB/s

Collecting asn1crypto>=0.21.0 (from cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)

100% |████████████████████████████████| 102kB 18kB/s

Requirement already satisfied: idna>=2.5 in d:\python37\lib\site-packages (from hyperlink>=17.1.1->Twisted>=13.1.0->Scrapy==1.6.0) (2.8)

Requirement already satisfied: setuptools in d:\python37\lib\site-packages (from zope.interface>=4.4.2->Twisted>=13.1.0->Scrapy==1.6.0) (39.0.1)

Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyOpenSSL->Scrapy==1.6.0)

Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)

100% |████████████████████████████████| 163kB 16kB/s

Building wheels for collected packages: PyDispatcher, pycparser

Building wheel for PyDispatcher (setup.py) ... done

Stored in directory: C:\Users\admin\AppData\Local\pip\Cache\wheels\88\99\96\cfef6665f9cb1522ee6757ae5955feedf2fe25f1737f91fa7f

Building wheel for pycparser (setup.py) ... done

Stored in directory: C:\Users\admin\AppData\Local\pip\Cache\wheels\f2\9a\90\de94f8556265ddc9d9c8b271b0f63e57b26fb1d67a45564511

Successfully built PyDispatcher pycparser

Installing collected packages: PyDispatcher, pycparser, cffi, asn1crypto, cryptography, pyOpenSSL, cssselect, w3lib, pyasn1, pyasn1-modules, service-identity, queuelib, parsel, Scrapy

Successfully installed PyDispatcher-2.0.5 Scrapy-1.6.0 asn1crypto-0.24.0 cffi-1.12.0 cryptography-2.5 cssselect-1.0.3 parsel-1.5.1 pyOpenSSL-19.0.0 pyasn1-0.4.5 pyasn1-modules-0.2.4 pycparser-2.19 queuelib-1.5.0 service-identity-18.1.0 w3lib-1.20.0