install scrapy

版本新一点的机器：

$rpm -qa | grep glibc glibc-common-2.12-1.80.1.alios6.x86_64
glibc-2.12-1.80.1.alios6.x86_64
glibc-headers-2.12-1.80.1.alios6.x86_64 glibc-2.12-1.80.1.alios6.i686
glibc-devel-2.12-1.80.1.alios6.x86_64

[admin@cssearch-1-1 /home/admin/scrapy_issues] $gcc --version gcc
(GCC) 4.4.7 20120313 (Red Hat 4.4.7-11) Copyright © 2010 Free
Software Foundation, Inc. This is free software; see the source for
copying conditions. There is NO warranty; not even for
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

老的机器，一直装不上，最终崩溃在glibc的版本上

$ rpm -qa | grep glibc glibc-devel-2.5-81 glibc-2.5-81
compat-glibc-2.3.4-2.26 glibc-devel-2.5-81 compat-glibc-2.3.4-2.26
glibc-headers-2.5-81 glibc-common-2.5-81
compat-glibc-headers-2.3.4-2.26 glibc-2.5-81

[kevin@cshadoop10 /home/kevin] $ gcc --version gcc (GCC) 4.1.2
20080704 (Red Hat 4.1.2-52) Copyright © 2006 Free Software
Foundation, Inc. This is free software; see the source for copying
conditions. There is NO warranty; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.

按scrapy官方文档安装即可
1, install python2.7
2, install pip
https://pip.pypa.io/en/latest/installing.html#install-pip

3, install scrapy
pip install Scrapy

WOW~~

Successfully installed Scrapy-1.0.3 Twisted-15.3.0 cffi-1.2.1
characteristic-14.3.0 cryptography-1.0 cssselect-0.9.1 enum34-1.0.4
idna-2.0 ipaddress-1.0.14 lxml-3.4.4 pyOpenSSL-0.15.1 pyasn1-0.1.8
pyasn1-modules-0.0.7 pycparser-2.14 queuelib-1.3.0
service-identity-14.0.0 six-1.9.0 w3lib-1.12.0 zope.interface-4.1.2

---

install scrapy

officinal doc:
http://doc.scrapy.org/en/latest/intro/install.html#intro-install-platform-notes

The installation steps assume that you have the following things
installed:

• Python 2.7
• pip and setuptools Python packages. Nowadays pip requires
• lxml. Most Linux distributions ships prepackaged versions of lxml. Otherwise refer to http://lxml.de/installation.html
• OpenSSL. This comes preinstalled in all operating systems, except Windows where the Python installer ships it bundled.

install pip

https://bootstrap.pypa.io/get-pip.py

install setuptools

$wget http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea --no-check-certificate
--2015-08-25 15:09:47--  http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
正在解析主机 pypi.python.org... 103.245.222.223
Connecting to pypi.python.org|103.245.222.223|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 301 Moved Permanently
位置：https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg [跟随至新的 URL]
--2015-08-25 15:09:47--  https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Connecting to pypi.python.org|103.245.222.223|:443... 已连接。
WARNING: cannot verify pypi.python.org's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended Validation Server CA':
  Unable to locally verify the issuer's authority.
WARNING: certificate common name `www.python.org' doesn't match requested host name `pypi.python.org'.
已发出 HTTP 请求，正在等待回应... 200 OK
长度：332005 (324K) [application/octet-stream]
Saving to: `setuptools-0.6c11-py2.7.egg'


100%[===========================================================================================================================================================>] 332,005      616K/s   in 0.5s    


2015-08-25 15:09:48 (616 KB/s) - `setuptools-0.6c11-py2.7.egg' saved [332005/332005]

python2.7 is not found
modified the

install lxml

sudo -H pip install lxml==3.4.2

  InsecurePlatformWarning
Collecting lxml==3.4.2
/usr/local/python27/lib/python2.7/site-packages/pip-7.0.3-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/python27/lib/python2.7/site-packages/pip-7.0.3-py2.7.egg/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Could not find a version that satisfies the requirement lxml==3.4.2 (from versions: )
No matching distribution found for lxml==3.4.2

refer:http://www.cnblogs.com/rwxwsblog/p/4557123.html

wget ftp://xmlsoft.org/libxml2/libxml2-git-snapshot.tar.gz

install open ssl

Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
No local packages or download links found for cryptography>=0.7
error: Could not find suitable distribution for Requirement.parse('cryptography>=0.7')

done

Using /usr/local/python27/lib/python2.7/site-packages
Finished processing dependencies for pyOpenSSL==0.15.1

install cryptography

Download error on https://pypi.python.org/simple/cffi/: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
Couldn't find index page for 'cffi' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
No local packages or download links found for cffi>=0.8
Traceback (most recent call last):
  File "setup.py", line 174, in <module>
    "test": PyTest,
  File "/usr/local/python27/lib/python2.7/distutils/core.py", line 112, in setup
    _setup_distribution = dist = klass(attrs)
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 268, in __init__
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 313, in fetch_build_eggs
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 830, in resolve
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1075, in best_match
  File "build/bdist.linux-x86_64/egg/pkg_resources/__init__.py", line 1087, in obtain
  File "build/bdist.linux-x86_64/egg/setuptools/dist.py", line 380, in fetch_build_egg
  File "build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line 623, in easy_install
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('cffi>=0.8')

    import _cffi_backend as backend
ImportError: libffi.so.6: cannot open shared object file: No such file or directory

fix set the lib path into /etc/ld.so.conf and sudo ldconfig

finally

Using /usr/local/python27/lib/python2.7/site-packages/cffi-1.2.1-py2.7-linux-x86_64.egg
Searching for pycparser==2.14
Best match: pycparser 2.14
Adding pycparser 2.14 to easy-install.pth file

Using /usr/local/python27/lib/python2.7/site-packages
Finished processing dependencies for cryptography==0.4

get v1.0 from https://pypi.python.org/pypi/cryptography/1.0

No local packages or download links found for ipaddress
error: Could not find suitable distribution for Requirement.parse('ipaddress')

get ipaddress from https://pypi.python.org/pypi/ipaddress/1.0.14

No local packages or download links found for enum34
error: Could not find suitable distribution for Requirement.parse('enum34')

…

No local packages or download links found for pyasn1>=0.1.8
error: Could not find suitable distribution for Requirement.parse('pyasn1>=0.1.8')

…

No local packages or download links found for idna>=2.0
error: Could not find suitable distribution for Requirement.parse('idna>=2.0')

done

Adding pycparser 2.14 to easy-install.pth file

Using /usr/local/python27/lib/python2.7/site-packages
Finished processing dependencies for cryptography==1.0

install cffi-1.2.1

gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/python27/include/python2.7 -c c/
_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
c/_cffi_backend.c:13:17: 错误：ffi.h：没有那个文件或目录

after install libffi

$sudo python setup.py build
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
running build
running build_py
running build_ext
building '_cffi_backend' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/python27/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/c/_cffi_backend.o -lffi -o build/lib.linux-x86_64-2.7/_cffi_backend.so

it seems that it’s ok but install fail

Download error on https://pypi.python.org/simple/: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found!
No local packages or download links found for pycparser
error: Could not find suitable distribution for Requirement.parse('pycparser')

after install pycparser

Installed /usr/local/python27/lib/python2.7/site-packages/cffi-1.2.1-py2.7-linux-x86_64.egg
Processing dependencies for cffi==1.2.1
Searching for pycparser==2.14
Best match: pycparser 2.14
Adding pycparser 2.14 to easy-install.pth file

Using /usr/local/python27/lib/python2.7/site-packages
Finished processing dependencies for cffi==1.2.1

install libffi

get the source from https://sourceware.org/libffi/

sudo cp /usr/local/lib/libffi-3.2.1/include/ffi* /usr/local/include/

install pycparser
get the source from https://pypi.python.org/pypi/pycparser/

finally
install scrapy

pip install Scrapy

so, get the source from github

No local packages or download links found for service-identity
error: Could not find suitable distribution for Requirement.parse('service-identity')

get from https://pypi.python.org/pypi/service_identity

No local packages or download links found for pyasn1-modules
error: Could not find suitable distribution for Requirement.parse('pyasn1-modules')

…

No local packages or download links found for characteristic>=14.0.0
error: Could not find suitable distribution for Requirement.parse('characteristic>=14.0.0')

done

Using /usr/local/python27/lib/python2.7/site-packages
Finished processing dependencies for service-identity==14.0.0

install scrapy again

No local packages or download links found for cssselect>=0.9
error: Could not find suitable distribution for Requirement.parse('cssselect>=0.9')

install libxslt

Crypto extensions will be available.
Enabling debugger
checking for libxml libraries >= 2.6.27... configure: error: Version 2.6.26 found. You need at least libxml2 2.6.27 for this version of libxslt