centos7 python3.7 scrapy安装

1、卸载无用版本，慎重操作

卸载python3
       rm -rf /usr/local/python3      卸载pyhton3
      whereis python3 |xargs rm -frv           删除所有残余文件
      成功卸载！
      whereis   python       查看现有安装的python

2、安装Python-3.7

1.安装编译相关工具
yum -y groupinstall "Development tools"
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
yum install libffi-devel -y
2.下载安装包解压
cd #回到用户目录
wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
tar -xvJf  Python-3.7.0.tar.xz
3.编译安装python
mkdir /usr/local/python3 #创建编译安装目录
cd Python-3.7.0
./configure --prefix=/usr/local/python3
make && make install
4.创建软连接
ln -s /usr/local/python3/bin/python3 /usr/local/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/local/bin/pip3
5.验证是否成功
python3 -V
pip3 -V

如遇到创建软连接报错，可能之前安装造成的文件已存在，删除之前的pip3文件，再重新创建软连接。

/usr/local/bin/pip3

3、安装scrapy

[root@hadron ~]# pip3 install scrapy
Collecting scrapy
  Downloading Scrapy-1.5.0-py2.py3-none-any.whl (251kB)
    100% |████████████████████████████████| 256kB 1.1MB/s 
Requirement already satisfied: lxml in /usr/local/python3/lib/python3.5/site-packages (from scrapy)
Collecting PyDispatcher>=2.0.5 (from scrapy)
  Downloading PyDispatcher-2.0.5.tar.gz
....
....
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=2.1.4->pyOpenSSL->scrapy)
  Downloading pycparser-2.18.tar.gz (245kB)
    100% |████████████████████████████████| 256kB 339kB/s 
Installing collected packages: PyDispatcher, zope.interface, constantly, incremental, six, attrs, Automat, hyperlink, Twisted, cssselect, w3lib, parsel, asn1crypto, pycparser, cffi, cryptography, pyOpenSSL, pyasn1, pyasn1-modules, service-identity, queuelib, scrapy
  Running setup.py install for PyDispatcher ... done
  Running setup.py install for Twisted ... done
  Running setup.py install for pycparser ... done
Successfully installed Automat-0.6.0 PyDispatcher-2.0.5 Twisted-17.9.0 asn1crypto-0.24.0 attrs-17.4.0 cffi-1.11.4 constantly-15.1.0 cryptography-2.1.4 cssselect-1.0.3 hyperlink-18.0.0 incremental-17.5.0 parsel-1.4.0 pyOpenSSL-17.5.0 pyasn1-0.4.2 pyasn1-modules-0.2.1 pycparser-2.18 queuelib-1.4.2 scrapy-1.5.0 service-identity-17.0.0 six-1.11.0 w3lib-1.19.0 zope.interface-4.4.3
[root@hadron ~]#

如果报错，升级pip版本后，再重新安装scrapy。

You are using pip version 10.0.1, however version 22.3.1 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

pip3 install --upgrade pip

创建软连接，如报错，删除 /usr/local/bin/scrapy  或者   /usr/bin/scrapy 目录下旧的文件，重新创建
ln -s /usr/local/python3/bin/scrapy  /usr/local/bin/scrapy

安装对应的包

//安装BeautifulSoup4
pip3 install BeautifulSoup4
pip3 install openpyxl
pip3 install pymysql

4、运行scrapy

cd /www/wwwroot/default/scarpy/
scrapy crawl xxxxx