Ubuntu16.04python3.6下安装Scrapy报错解决办法

首先安装scrapy是没问题的，但是当导入模块时就报错了，具体如下：

root@ubuntu:/usr/bin# pip3 install scrapy

Collecting scrapy
  Downloading https://files.pythonhosted.org/packages/db/9c/cb15b2dc6003a805afd21b9b396e0e965800765b51da72fe17cf340b9be2/Scrapy-1.5.0-py2.py3-none-any.whl (251kB)
    100% |████████████████████████████████| 256kB 22kB/s 
Collecting cssselect>=0.9 (from scrapy)
  Downloading https://files.pythonhosted.org/packages/7b/44/25b7283e50585f0b4156960691d951b05d061abf4a714078393e51929b30/cssselect-1.0.3-py2.py3-none-any.whl
Collecting pyOpenSSL (from scrapy)
  Downloading https://files.pythonhosted.org/packages/79/db/7c0cfe4aa8341a5fab4638952520d8db6ab85ff84505e12c00ea311c3516/pyOpenSSL-17.5.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 17kB/s 
Collecting lxml (from scrapy)
  Downloading https://files.pythonhosted.org/packages/a7/b9/ccf46cea0f698b40bca2a9c1a44039c336fe1988b82de4f7353be7a8396a/lxml-4.2.1-cp36-cp36m-manylinux1_x86_64.whl (5.6MB)
    100% |████████████████████████████████| 5.7MB 99kB/s 
Collecting six>=1.5.2 (from scrapy)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting service-identity (from scrapy)
  Downloading https://files.pythonhosted.org/packages/29/fa/995e364220979e577e7ca232440961db0bf996b6edaf586a7d1bd14d81f1/service_identity-17.0.0-py2.py3-none-any.whl
Collecting queuelib (from scrapy)
  Downloading https://files.pythonhosted.org/packages/4c/85/ae64e9145f39dd6d14f8af3fa809a270ef3729f3b90b3c0cf5aa242ab0d4/queuelib-1.5.0-py2.py3-none-any.whl
Collecting parsel>=1.1 (from scrapy)
  Downloading https://files.pythonhosted.org/packages/bc/b4/2fd37d6f6a7e35cbc4c2613a789221ef1109708d5d4fb9fd5f6f721a43c9/parsel-1.4.0-py2.py3-none-any.whl
Requirement already satisfied: Twisted>=13.1.0 in ./python3.6/lib/python3.6/site-packages/Twisted-15.2.1-py3.6.egg (from scrapy) (15.2.1)
Collecting w3lib>=1.17.0 (from scrapy)
  Downloading https://files.pythonhosted.org/packages/37/94/40c93ad0cadac0f8cb729e1668823c71532fd4a7361b141aec535acb68e3/w3lib-1.19.0-py2.py3-none-any.whl
Collecting PyDispatcher>=2.0.5 (from scrapy)
  Downloading https://files.pythonhosted.org/packages/cd/37/39aca520918ce1935bea9c356bcbb7ed7e52ad4e31bff9b943dfc8e7115b/PyDispatcher-2.0.5.tar.gz
Collecting cryptography>=2.1.4 (from pyOpenSSL->scrapy)
  Downloading https://files.pythonhosted.org/packages/fa/f4/3cde3604972dfa2b0fea85b9711948bb4fb70ab64095322aef35071bd254/cryptography-2.2.2-cp34-abi3-manylinux1_x86_64.whl (2.2MB)
    100% |████████████████████████████████| 2.2MB 51kB/s 
Collecting pyasn1 (from service-identity->scrapy)
  Downloading https://files.pythonhosted.org/packages/ba/fe/02e3e2ee243966b143657fb8bd6bc97595841163b6d8c26820944acaec4d/pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 83kB/s 
Collecting attrs (from service-identity->scrapy)
  Downloading https://files.pythonhosted.org/packages/41/59/cedf87e91ed541be7957c501a92102f9cc6363c623a7666d69d51c78ac5b/attrs-18.1.0-py2.py3-none-any.whl
Collecting pyasn1-modules (from service-identity->scrapy)
  Downloading https://files.pythonhosted.org/packages/e9/51/bcd96bf6231d4b2cc5e023c511bee86637ba375c44a6f9d1b4b7ad1ce4b9/pyasn1_modules-0.2.1-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 105kB/s 
Requirement already satisfied: zope.interface>=4.0.2 in ./python3.6/lib/python3.6/site-packages/zope.interface-4.5.0-py3.6-linux-x86_64.egg (from Twisted>=13.1.0->scrapy) (4.5.0)
Collecting cffi>=1.7; platform_python_implementation != "PyPy" (from cryptography>=2.1.4->pyOpenSSL->scrapy)
  Downloading https://files.pythonhosted.org/packages/6d/c0/47db8f624f3e4e2f3f27be03a93379d1ba16a1450a7b1aacfa0366e2c0dd/cffi-1.11.5-cp36-cp36m-manylinux1_x86_64.whl (421kB)
    100% |████████████████████████████████| 430kB 94kB/s 
Collecting idna>=2.1 (from cryptography>=2.1.4->pyOpenSSL->scrapy)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 50kB/s 
Collecting asn1crypto>=0.21.0 (from cryptography>=2.1.4->pyOpenSSL->scrapy)
  Downloading https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)
    100% |████████████████████████████████| 102kB 41kB/s 
Requirement already satisfied: setuptools in ./python3.6/lib/python3.6/site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->scrapy) (39.0.1)
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->pyOpenSSL->scrapy)
  Downloading https://files.pythonhosted.org/packages/8c/2d/aad7f16146f4197a11f8e91fb81df177adcc2073d36a17b1491fd09df6ed/pycparser-2.18.tar.gz (245kB)
    100% |████████████████████████████████| 256kB 42kB/s 
Installing collected packages: cssselect, pycparser, cffi, idna, asn1crypto, six, cryptography, pyOpenSSL, lxml, pyasn1, attrs, pyasn1-modules, service-identity, queuelib, w3lib, parsel, PyDispatcher, scrapy
  Running setup.py install for pycparser ... done
  Running setup.py install for PyDispatcher ... done
  The script scrapy is installed in '/usr/bin/python3.6/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed PyDispatcher-2.0.5 asn1crypto-0.24.0 attrs-18.1.0 cffi-1.11.5 cryptography-2.2.2 cssselect-1.0.3 idna-2.6 lxml-4.2.1 parsel-1.4.0 pyOpenSSL-17.5.0 pyasn1-0.4.2 pyasn1-modules-0.2.1 pycparser-2.18 queuelib-1.5.0 scrapy-1.5.0 service-identity-17.0.0 six-1.11.0 w3lib-1.19.0

貌似安装成功了，来测试一下：

root@ubuntu:/usr/bin# python
Python 3.6.5 (default, May 12 2018, 19:37:55) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/bin/python3.6/lib/python3.6/site-packages/scrapy/__init__.py", line 27, in <module>
    from . import _monkeypatches
  File "/usr/bin/python3.6/lib/python3.6/site-packages/scrapy/_monkeypatches.py", line 20, in <module>
    import twisted.persisted.styles  # NOQA
ModuleNotFoundError: No module named 'twisted.persisted'
>>>

报错了，ModuleNotFoundError: No module named ‘twisted.persisted’

pip3 install twisted==13.1.0

执行上述命令，还是报错了。。。
**Could not find a version that satisfies the requirement Twisted>=13.1.0 (from Scrapy) (from versions: )
No matching distribution found for Twisted>=13.1.0 (from Scrapy)**
网上查到原因说是没有安装Twisted，于是乎执行了如下命令

wget https://pypi.python.org/packages/source/T/Twisted/Twisted-15.2.1.tar.bz2
tar -xjvf Twisted-15.2.1.tar.bz2
cd Twisted-15.2.1
python setup.py install

成功后，进入python导入scrapy还是提示如上错误，原因还是Twisted版本问题，之前使用的是13.1.0,后面安装了15.2.1版本，还是有问题，于是乎使用高版本的17.1.0Twisted即可.安装包连接如下所示，使用Ubuntu命令即可安全下载：https://twistedmatrix.com/Releases/Twisted/17.1/Twisted-17.1.0.tar.bz2

root@ubuntu:/usr#wget https://twistedmatrix.com/Releases/Twisted/17.1/Twisted-17.1.0.tar.bz2  
root@ubuntu:/usr#tar -jxvf Twisted-17.1.0.tar.bz2  
root@ubuntu:/usr#cd Twisted-17.1.0  
root@ubuntu:/usr#python setup.py install  
root@ubuntu:/usr#cd ..  
root@ubuntu:/usr# pip3 install scrapy  
Requirement already satisfied: scrapy in ./bin/python3.6/lib/python3.6/site-packages (1.5.0)
Requirement already satisfied: pyOpenSSL in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (17.5.0)
Requirement already satisfied: cssselect>=0.9 in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (1.0.3)
Requirement already satisfied: six>=1.5.2 in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (1.11.0)
Requirement already satisfied: PyDispatcher>=2.0.5 in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (2.0.5)
Requirement already satisfied: lxml in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (4.2.1)
Requirement already satisfied: queuelib in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (1.5.0)
Requirement already satisfied: service-identity in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (17.0.0)
Requirement already satisfied: w3lib>=1.17.0 in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (1.19.0)
Requirement already satisfied: Twisted>=13.1.0 in ./bin/python3.6/lib/python3.6/site-packages/Twisted-17.1.0-py3.6-linux-x86_64.egg (from scrapy) (17.1.0)
Requirement already satisfied: parsel>=1.1 in ./bin/python3.6/lib/python3.6/site-packages (from scrapy) (1.4.0)
Requirement already satisfied: cryptography>=2.1.4 in ./bin/python3.6/lib/python3.6/site-packages (from pyOpenSSL->scrapy) (2.2.2)
Requirement already satisfied: attrs in ./bin/python3.6/lib/python3.6/site-packages (from service-identity->scrapy) (18.1.0)
Requirement already satisfied: pyasn1-modules in ./bin/python3.6/lib/python3.6/site-packages (from service-identity->scrapy) (0.2.1)
Requirement already satisfied: pyasn1 in ./bin/python3.6/lib/python3.6/site-packages (from service-identity->scrapy) (0.4.2)
Requirement already satisfied: zope.interface>=4.0.2 in ./bin/python3.6/lib/python3.6/site-packages/zope.interface-4.5.0-py3.6-linux-x86_64.egg (from Twisted>=13.1.0->scrapy) (4.5.0)
Requirement already satisfied: constantly>=15.1 in ./bin/python3.6/lib/python3.6/site-packages/constantly-15.1.0-py3.6.egg (from Twisted>=13.1.0->scrapy) (15.1.0)
Requirement already satisfied: incremental>=16.10.1 in ./Twisted-17.1.0/.eggs/incremental-17.5.0-py3.6.egg (from Twisted>=13.1.0->scrapy) (17.5.0)
Requirement already satisfied: Automat>=0.3.0 in ./bin/python3.6/lib/python3.6/site-packages/Automat-0.6.0-py3.6.egg (from Twisted>=13.1.0->scrapy) (0.6.0)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != "PyPy" in ./bin/python3.6/lib/python3.6/site-packages (from cryptography>=2.1.4->pyOpenSSL->scrapy) (1.11.5)
Requirement already satisfied: idna>=2.1 in ./bin/python3.6/lib/python3.6/site-packages (from cryptography>=2.1.4->pyOpenSSL->scrapy) (2.6)
Requirement already satisfied: asn1crypto>=0.21.0 in ./bin/python3.6/lib/python3.6/site-packages (from cryptography>=2.1.4->pyOpenSSL->scrapy) (0.24.0)
Requirement already satisfied: setuptools in ./bin/python3.6/lib/python3.6/site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->scrapy) (39.0.1)
Requirement already satisfied: pycparser in ./bin/python3.6/lib/python3.6/site-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->pyOpenSSL->scrapy) (2.18)
root@ubuntu:/usr# python
Python 3.6.5 (default, May 12 2018, 19:37:55) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>>

如上所示，终于将scrapy安装成功了，实在不容易啊！