Windows安装scrapy报错解决办法

Windows安装scrapy报错解决办法

在Windows系统中，如果我们直接使用pip install scrapy 进行安装会报一个错误

报错信息如下：
Using legacy 'setup.py install' for Twisted, since package 'wheel' is not installed.
Using legacy 'setup.py install' for protego, since package 'wheel' is not installed.
Using legacy 'setup.py install' for PyDispatcher, since package 'wheel' is not installed.
Installing collected packages: pycparser, cffi, cryptography, attrs, pyasn1, pyasn1-modules, service-identity, zope.interface, pyOpenSSL, constantly, increm
ental, Automat, hyperlink, PyHamcrest, Twisted, itemadapter, cssselect, w3lib, parsel, protego, queuelib, jmespath, itemloaders, PyDispatcher, scrapy
    Running setup.py install for Twisted ... error
    ERROR: Command errored out with exit status 1:
     command: 'd:\python_projects\lemtest\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\PC\\AppData\\Local
\\Temp\\pip-install-fex100n2\\twisted\\setup.py'"'"'; __file__='"'"'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-install-fex100n2\\twisted\\setup.py'"'"';f=geta
ttr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' i
nstall --record 'C:\Users\PC\AppData\Local\Temp\pip-record-y7lsfj81\install-record.txt' --single-version-externally-managed --compile --install-headers 'd:\
python_projects\lemtest\venv\include\site\python3.8\Twisted'
         cwd: C:\Users\PC\AppData\Local\Temp\pip-install-fex100n2\twisted\
......中间省略若干
running build_ext
    building 'twisted.test.raiser' extension
    error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-to
ols/
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'd:\python_projects\lemtest\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '
"'"'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-install-fex100n2\\twisted\\setup.py'"'"'; __file__='"'"'C:\\Users\\PC\\AppData\\Local\\Temp\\pip-install-fex100
n2\\twisted\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(co
de, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\PC\AppData\Local\Temp\pip-record-y7lsfj81\install-record.txt' --single-version-externally-managed
 --compile --install-headers 'd:\python_projects\lemtest\venv\include\site\python3.8\Twisted' Check the logs for full command output.

解决上面的错误，我们需要按照如下步骤：

1. 在https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml 中下载对应Python版本与系统的 Twisted 包，然后用pip 进行安装
   

2. 在https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml 中下载对应Python版本与系统的 lxml 包，然后用pip进行安装
   

3. 使用 pip install scrapy 进行安装 scrapy ,安装完成后在命令行输入scrapy检查安装是否成功，若未报错，则安装成功

# 安装lxml
(venv) D:\Python_Projects>pip install  d:\download\lxml-4.6.1-cp38-cp38-win_amd64.whl
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Processing d:\python_projects\lemtest\lxml-4.6.1-cp38-cp38-win_amd64.whl
Installing collected packages: lxml
Successfully installed lxml-4.6.1

# 安装Twisted
(venv) D:\Python_Projects>pip install  d:\download\Twisted-20.3.0-cp38-cp38-win_amd64.whl
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Processing d:\python_projects\lemtest\twisted-20.3.0-cp38-cp38-win_amd64.whl
Requirement already satisfied: zope.interface>=4.4.2 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (5.1.2)
Requirement already satisfied: Automat>=0.3.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (20.2.0)
Requirement already satisfied: constantly>=15.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (15.1.0)
Requirement already satisfied: hyperlink>=17.1.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (20.0.1)
Requirement already satisfied: attrs>=19.2.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (20.2.0)
Requirement already satisfied: PyHamcrest!=1.10.0,>=1.9.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (2.0.2)
Requirement already satisfied: incremental>=16.10.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted==20.3.0) (17.5.0)
Requirement already satisfied: setuptools in d:\python_projects\lemtest\venv\lib\site-packages (from zope.interface>=4.4.2->Twisted==20.3.0) (50.3.2)
Requirement already satisfied: six in d:\python_projects\lemtest\venv\lib\site-packages (from Automat>=0.3.0->Twisted==20.3.0) (1.15.0)
Requirement already satisfied: idna>=2.5 in d:\python_projects\lemtest\venv\lib\site-packages (from hyperlink>=17.1.1->Twisted==20.3.0) (2.10)
Installing collected packages: Twisted
Successfully installed Twisted-20.3.0

# 安装scrapy
(venv) D:\Python_Projects>pip install scrapy
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Collecting scrapy
  Using cached https://mirrors.aliyun.com/pypi/packages/fd/49/425c39549e277510d5cbc9f8f6ae36e1d4faf9fd768b613af1dd6b786c60/Scrapy-2.4.0-py2.py3-none-any.whl
 (239 kB)
Requirement already satisfied: protego>=0.1.15 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (0.1.16)
Requirement already satisfied: pyOpenSSL>=16.2.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (19.1.0)
Requirement already satisfied: service-identity>=16.0.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (18.1.0)
Requirement already satisfied: cssselect>=0.9.1 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.1.0)
Requirement already satisfied: Twisted>=17.9.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (20.3.0)
Requirement already satisfied: parsel>=1.5.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.6.0)
Requirement already satisfied: queuelib>=1.4.2 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.5.0)
Requirement already satisfied: w3lib>=1.17.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.22.0)
Requirement already satisfied: itemadapter>=0.1.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (0.1.1)
Requirement already satisfied: lxml>=3.5.0; platform_python_implementation == "CPython" in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (
4.6.1)
Requirement already satisfied: itemloaders>=1.0.1 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (1.0.3)
Requirement already satisfied: PyDispatcher>=2.0.5 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (2.0.5)
Requirement already satisfied: zope.interface>=4.1.3 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (5.1.2)
Requirement already satisfied: cryptography>=2.0 in d:\python_projects\lemtest\venv\lib\site-packages (from scrapy) (3.2)
Requirement already satisfied: six in d:\python_projects\lemtest\venv\lib\site-packages (from protego>=0.1.15->scrapy) (1.15.0)
Requirement already satisfied: pyasn1 in d:\python_projects\lemtest\venv\lib\site-packages (from service-identity>=16.0.0->scrapy) (0.4.8)
Requirement already satisfied: attrs>=16.0.0 in d:\python_projects\lemtest\venv\lib\site-packages (from service-identity>=16.0.0->scrapy) (20.2.0)
Requirement already satisfied: pyasn1-modules in d:\python_projects\lemtest\venv\lib\site-packages (from service-identity>=16.0.0->scrapy) (0.2.8)
Requirement already satisfied: hyperlink>=17.1.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (20.0.1)
Requirement already satisfied: incremental>=16.10.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (17.5.0)
Requirement already satisfied: constantly>=15.1 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (15.1.0)
Requirement already satisfied: Automat>=0.3.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (20.2.0)
Requirement already satisfied: PyHamcrest!=1.10.0,>=1.9.0 in d:\python_projects\lemtest\venv\lib\site-packages (from Twisted>=17.9.0->scrapy) (2.0.2)
Requirement already satisfied: jmespath>=0.9.5 in d:\python_projects\lemtest\venv\lib\site-packages (from itemloaders>=1.0.1->scrapy) (0.10.0)
Requirement already satisfied: setuptools in d:\python_projects\lemtest\venv\lib\site-packages (from zope.interface>=4.1.3->scrapy) (50.3.2)
Requirement already satisfied: cffi!=1.11.3,>=1.8 in d:\python_projects\lemtest\venv\lib\site-packages (from cryptography>=2.0->scrapy) (1.14.3)
Requirement already satisfied: idna>=2.5 in d:\python_projects\lemtest\venv\lib\site-packages (from hyperlink>=17.1.1->Twisted>=17.9.0->scrapy) (2.10)
Requirement already satisfied: pycparser in d:\python_projects\lemtest\venv\lib\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.0->scrapy) (2.20)
Installing collected packages: scrapy
Successfully installed scrapy-2.4.0

# 验证是否安装成功
(venv) D:\Python_Projects>scrapy
Scrapy 2.4.0 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  commands
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command