今天给线上服务器装爬虫环境,顺便记录下安装过程,网上有很多类似的安装过程,我只是整理+验证,希望对需要安装的人有帮助
pyOpenSSL安装
yum install gcc libffi-devel python-devel openssl-devel
yum install openssl-devel python-devel python-sphinx
下载地址:
https://pypi.python.org/pypi/pyOpenSSL/0.15.1/ pyOpenSSL-0.15.1.tar.gz
linux 上如果不成功,多执行几次easy_install
easy_install https://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.15.1.tar.gz#md5=f447644afcbd5f0a1f47350fec63a4c6
安装python
安装python
wget https://www.python.org/ftp/python/2.7.11/Python-2.7.11.tgz
tar zxvf Python-2.7.11.tgz
cd Python-2.7.11
./configure --prefix=/usr/local
make && make altinstall
检查Python版本
python -V
问题:
configure: error: no acceptable C compiler found in $PATH 问题解决
解决:yum install gcc
如果已有旧版本:
查看安装目录:whereis python2.7
不出意外,新版本应该安装在:/usr/local/bin/python2.7
**查看新版本:/usr/local/bin/python2.7 -V
而系统的默认python放置在/usr/bin/python2.6处(这个2.6是系统默认版本)
因此此时需要建立软连接指向当前系统默认的python目录处,让系统使用新的python版本
mv /usr/bin/python /usr/bin/python2.6 //对系统默认版本python进行操作
ln -s /usr/local/bin/python2.7 /usr/bin/python //注意第一个路径是新安装的python版本路径,而第二个路径是系统默认路径**
修改完成,此时输入python -V或者python就看看到版本号已经修改到了新安装的版本
安装scrapy
安装python-devel
yum search all python-devel
yum install python-devel.x86_64
运行yum报错:No module named yum
1、查看已安装python的版本,可能是当前系统存在多个python导致
[root@pachong2 Python-2.7.11]# whereis python
python: /usr/bin/python2.6-config /usr/bin/python2.6 /usr/bin/python /usr/lib/python2.6 /usr/lib64/python2.6 /usr/local/bin/python2.7-config /usr/local/bin/python2.7 /usr/local/bin/python /usr/local/lib/python2.7 /usr/include/python2.6 /usr/share/man/man1/python.1.gz
[root@test ~]# vi /usr/bin/yum
将 #!/usr/bin/python 修改为 #!/usr/bin/python2.6
安装setuptools
wget --no-check-certificate
https://bootstrap.pypa.io/ez_setup.py
–2016-05-10 10:48:35– https://bootstrap.pypa.io/ez_setup.py
Resolving bootstrap.pypa.io… 103.245.222.175
Connecting to bootstrap.pypa.io|103.245.222.175|:443… connected.
WARNING: certificate common name “*.c.ssl.fastly.net” doesn’t match requested host name “bootstrap.pypa.io”.
HTTP request sent, awaiting response… 200 OK
Length: 12385 (12K) [text/x-python]
Saving to: “ez_setup.py”
100%[==========================================================================================================>] 12,385 –.-K/s in 0s
2016-05-10 10:48:35 (105 MB/s) - “ez_setup.py” saved [12385/12385]
ll
total 83956
-rw-r–r– 1 root root 12385 May 6 21:30 ez_setup.py
python ez_setup.py --insecure
Downloading https://pypi.io/packages/source/s/setuptools/setuptools-21.0.0.zip
Extracting in /tmp/tmp9OQvdP
Now working in /tmp/tmp9OQvdP/setuptools-21.0.0
Installing Setuptools
running install
running bdist_egg
running egg_info
writing requirements to setuptools.egg-info/requires.txt
…………….
Copying setuptools-21.0.0-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding setuptools 21.0.0 to easy-install.pth file
Installing easy_install script to /usr/local/bin
Installing easy_install-2.7 script to /usr/local/bin
Installed /usr/local/lib/python2.7/site-packages/setuptools-21.0.0-py2.7.egg
Processing dependencies for setuptools==21.0.0
Finished processing dependencies for setuptools==21.0.0
报错:
urllib2.URLError:
解决:http://blog.csdn.net/zyz511919766/article/details/25049365
pip安装
- 下载文件
注:如果第一次执行没有下载下来,可以多执行几次wget
wget https://bootstrap.pypa.io/get-pip.py --no-check-certificate
- 执行安装
python get-pip.py
- 查看版本
pip –V
zope.interface安装
网页地址:https://pypi.python.org/pypi/zope.interface/4.1.3#downloads
wget --no-check-certificate https://pypi.python.org/packages/9d/81/2509ca3c6f59080123c1a8a97125eb48414022618cec0e64eb1313727bfe/zope.interface-4.1.3.tar.gz#md5=9ae3d24c0c7415deb249dd1a132f0f79
tar -zxvf zope.interface-4.1.3.tar.gz
cd zope.interface-4.1.3
python setup.py build
python setup.py install
安装完成后,可以在/usr/local/lib/python2.7/site-packages下面看到
zope.interface-4.1.3-py2.7-linux-x86_64.egg
安装openssl
wget https://www.openssl.org/source/openssl-1.0.2h.tar.gz
tar -zxvf openssl-1.0.2h.tar.gz
cd openssl-1.0.2h
./config
make
make test
make install
查看openssl的版本
openssl version
若还是旧版的话,执行下面的命令:
mv /usr/bin/openssl /root/
ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl
再查看版本
openssl version
OpenSSL 1.0.2h 3 May 2016
Twisted安装
wget http://twistedmatrix.com/Releases/Twisted/12.0/Twisted-12.0.0.tar.bz2
bzip2 -d Twisted-12.0.0.tar.bz2
tar -xvf Twisted-12.0.0.tar
cd Twisted-12.0.0
python setup.py install
W3lib安装
wget --no-check-certificate http://pypi.python.org/packages/source/w/w3lib/w3lib-1.0.tar.gz
wget http://pypi.python.org/packages/source/w/w3lib/w3lib-1.0.tar.gz
[root@localhost /opt/w3lib-1.0]$ tar -xvzf w3lib-1.0.tar.gz
[root@localhost /opt/w3lib-1.0]$ cd w3lib-1.0
[root@localhost /opt/w3lib-1.0]$ python setup.py install
安装完成后,可以在/usr/local/lib/python2.7/site-packages下面看到w3lib和w3lib-1.0-py2.7.egg-info。
安装libxml2
下载地址如下所示:
http://download.chinaunix.net/down.php?id=28497&ResourceID=6095&site=1
libxml2-2.7.4.tar.gz
注:如果需要,安装rzsz命令:
yum -y install lrzsz
cd libxml2-2.7.4
./configure
make
make install
需要python中import libxml2成功,则
cd python
python setup.py install
注:
编译libxml2时报错,./.libs/libxml2.so: undefined reference to `gzopen64’ :
1,下载zlib-1.2.3.tar.gz,解压,configure后,vi Makefile ,然后在CFLAGS 后面添加 -fPIC 参数,编译make然后make install
2, 重新编译libxml2,
./configure –prefix=/usr/local/libxml2 –with-zlib=/usr/local/zlib/
然后vi Makefile ,然后在CFLAGS 后面添加 -fPIC 参数,编译make然后make install
ok
lxml安装
yum search all libxslt-devel
yum install libxslt-devel.x86_64
pip install lxml
安装scrapy
wget --no-check-certificate https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz#md5=cefb981076a26112d6f9b76cb7e5ba7d
tar -xvzf Scrapy-1.0.3.tar.gz
cd Scrapy-1.0.3
python setup.py install
报错:
error: The ‘pyasn1’ distribution was not found and is required by service-identity
这个错误只遇到一次,重新执行“python setup.py install”安装成功。
注:网上有说pip install pyasn1,pip install cryptography 安装这两个东西,我重新执行安装命令就成功了,所以并没有装pyasn1和cryptography。
安装redis
- 安装需要的支持环境
在安装Redis之前首要先做的是安装Unix的Tcl工具,如果不安装的话后期将无法对Redis进行测试。在后期执行make test的时候返回如下错误信息:You need tcl 8.xuyao de5 or newer in order to run the Redis test,具体的流程为:
复制代码代码如下:
cd /usr/local/src
wget http://downloads.sourceforge.net/tcl/tcl8.6.3-src.tar.gz
tar -zxvf tcl8.6.3-src.tar.gz
cd tcl8.6.3/unix/
./configure
make
make install
- 安装redis
安装redis的过程非常的简单,具体教程官网也有。具体如下:http://redis.io/download
复制代码代码如下:
cd /usr/local/src
$ wget http://download.redis.io/releases/redis-3.0.5.tar.gz
tar zxvf redis-3.0.5.tar.gz
cd redis-3.0.5
make
make PREFIX=/usr/local/redis install
其中PREFIX=/usr/local/redis可以省略,省略情况下redis会默认安装到/usr/local/bin目录下。
3. 测试Redis
代码如下:
cd src
make test
通过以上命令就要可以对redis进行加大的测试。
问题:
在执行make test时报错:
You need tcl 8.5 or newer in order to run the Redis test
执行如下解决:
wget http://downloads.sourceforge.net/tcl/tcl8.6.1-src.tar.gz
sudo tar xzvf tcl8.6.1-src.tar.gz -C /usr/local/
cd /usr/local/tcl8.6.1/unix/
./configure
make
make install
继续报错:
* [err]: Test replication partial resync: ok psync (diskless: yes, reconnect: 1) in tests/integration/replication-psync.tcl
Expected condition ‘[s -1 sync_partial_ok] > 0’ to be true ([s -1 sync_partial_ok] > 0)
Cleanup: may take some time… OK
解决办法:
更改redis安装目录下, tests/integration/replication-psync.tcl 文件:
vi tests/integration/replication-psync.tcl
把对应报错的那段代码中的 after后面的数字,从100改成 500。我个人觉得,这个参数貌似是等待的毫秒数
test "Test replication partial resync: $descr (diskless: $diskless, reconnect: $reconnect)" {
# Now while the clients are writing data, break the maste-slave
# link multiple times.
if ($reconnect) {
for {set j 0} {$j < $duration*10} {incr j} {
after 500
# catch {puts "MASTER [$master dbsize] keys, SLAVE [$slave dbsize] keys"}
if {($j % 20) == 0} {
catch {
if {$delay} {
解决问题。
- 查看安装版本:
redis-server --version
mysql安装
安装mysql数据库
http://www.cnblogs.com/xiaoluo501395377/archive/2013/04/07/3003278.html
安装mysql-python
pip install mysql-python
验证:
>>> import MySQLdb
>>>
成功!