scrapy安装指南

最新推荐文章于 2020-03-25 22:39:29 发布

转载最新推荐文章于 2020-03-25 22:39:29 发布 · 1.9k 阅读

文章标签：

#python

python 专栏收录该内容

10 篇文章

订阅专栏

windows 7/8 安装scrapy

#官方文档有些问题，下述安装教程通过实践旧版安装教程

windows全部安装包下载：

https://github.com/addwork/scrapy-windows

1.安装python2.7，将 C:\python27和C:\python27\Scripts添加至环境变量

详细参考：python与工具pip安装使用熟悉python可以安装pip，不熟悉安装配置好python后继续。

2.安装OpenSSL，将openssl路径添(如 D:\openssl-win32\bin)添加至环境变量

依赖VC++ 的编译工具，如果安装了vc++/vs2008/vs2010/vs2012的可以直接安装OpenSSL，否则需要安装VC++目录下的执行文件，VC++2012是针对win8，win7的2012/2008均可.安装完成可执行文件即可安装OpenSSL.

3.安装包内可执行文件

pywin32，Twisted，zope.interfase，libxml，libxml2，pyOpenSSL

4. 其他压缩文件安装

解压w3lib-1.5.tar.gz,libxml-2.9.1.tar.gz,scrapy-0.22.tar.gz至D：盘

D:>cd w3lib-1.5
D:\w3lib-1.5>python setup.py install
D:\w3lib-1.5>cd d/libxslt
D:\libxslt>python setup.py install
D:\libxslt>cd d/scrapy-0.22 
D:\scrapy-0.22>python setup.py install 
D:\scrapy-0.22>python
>>>import w3lib

D:>scrapy -h
Scrapy 0.22.2 - no active project
Usage:
  scrapy

windows安装补充:

如果pip安装成功,可以使用pip安装scrapy（openssl和pywin32要单独安装）

pip install Zope.Interface
pip install Twisted
pip install w3lib
pip install libxml2
pip install pyopenssl
pip install scrapy

Ubuntu

$sudo apt-get install gcc
$sudo apt-get install libxml2-dev python-dev libxslt1-dev libffi-dev python-lxml libffi-dev
$sudo pip install service_identity
$sudo pip install Scrapy

Fedora

$wget https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py 
$python get-pip.py
$rpm [Sisyphus] http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus x86_64 classic 
$rpm [Sisyphus] http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus noarch classic 
$sudo yum update 
$sudo yum install libxslt-devel libxml2-devel python-devel
$sudo yum install libffi libffi-devel
$sudo yum install python-setuptools
$sudo yum install pyOpenSSL 
$sudo yum install gcc 
$sudo pip install scrapy 
$sudo pip install service_identity

Centos

centos install python2.7

$yum -y update
$yum gruopinstall -y development
$yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel
$wget http://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
$tar -zxvf Python-2.7.6.tar.xz
$cd Python-2.7.6
$./configure
$make && make altinstall
$mv /usr/bin/python /usr/bin/python2.6.6
$ln -s /usr/local/bin/python2.7 /usr/bin/python
$vi /usr/bin/yum
  del #!/usr/bin/python
       #!/usr/bin/python2.6.6
$yum install curl
$curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$python get-pip.py
$yum install libxslt-devel
$pip install scrapy

$scrapy -h
Scrapy 0.22.0 - no active project
Usage:
  scrapy