原文地址http://www.cnblogs.com/txw1958/archive/2012/07/12/scrapy_installation_introduce.html
二、 安装Python2.7
下载地址:http://www.python.org/ftp/python/2.7.3/python-2.7.3.msi
1) 安装python
安装目录:D:\Python27
2) 添加环境变量
略System Properties -> Advanced -> Environment Variables - >System Variables -> Path -> Edit
3) 验证环境变量
T:\>set Path Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;D:\Rational\common;D:\Rational\ClearCase\bin;D:\Python27;D:\Python27\Scripts PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH
4) 验证Python
T:\>python Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> exit() T:\>
三、 安装Twisted
Twisted is an event-driven networking engine written in Python and licensed under the open source
1) 安装setuptools
Download, build, install, upgrade, and uninstall Python packages -- easily!
官方主页:http://pypi.python.org/pypi/setuptools
下载地址:http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11.win32-py2.7.exe
安装过程:略
2) 安装Zope.Interface
官方主页:http://pypi.python.org/pypi/zope.interface/
下载地址:http://pypi.python.org/packages/2.7/z/zope.interface/zope.interface-4.0.1-py2.7-win32.egg
安装过程:
T:\>d: D:\>cd D:\Python27\Scripts D:\Python27\Scripts>easy_install.exe zope.interface-4.0.1-py2.7-win32.egg Processing zope.interface-4.0.1-py2.7-win32.egg creating d:\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.egg Extracting zope.interface-4.0.1-py2.7-win32.egg to d:\python27\lib\site-packages Adding zope.interface 4.0.1 to easy-install.pth file Installed d:\python27\lib\site-packages\zope.interface-4.0.1-py2.7-win32.egg Processing dependencies for zope.interface==4.0.1 Finished processing dependencies for zope.interface==4.0.1 D:\Python27\Scripts>
验证安装:
D:\Python27\Scripts>python Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import zope.interface >>>
3) 安装Twisted
官方主页:http://twistedmatrix.com/trac/wiki/TwistedProject
下载地址:http://pypi.python.org/packages/2.7/T/Twisted/Twisted-12.1.0.win32-py2.7.msi
安装过程:略
四、 安装w3lib
官方主页:http://pypi.python.org/pypi/w3lib
下载地址: http://pypi.python.org/packages/source/w/w3lib/w3lib-1.2.tar.gz
解压过程:略
安装过程:
T:\w3lib-1.2>python setup.py install running install running build running build_py creating build creating build\lib creating build\lib\w3lib copying w3lib\encoding.py -> build\lib\w3lib copying w3lib\form.py -> build\lib\w3lib copying w3lib\html.py -> build\lib\w3lib copying w3lib\http.py -> build\lib\w3lib copying w3lib\url.py -> build\lib\w3lib copying w3lib\util.py -> build\lib\w3lib copying w3lib\__init__.py -> build\lib\w3lib running install_lib creating D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\encoding.py -> D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\form.py -> D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\html.py -> D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\http.py -> D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\url.py -> D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\util.py -> D:\Python27\Lib\site-packages\w3lib copying build\lib\w3lib\__init__.py -> D:\Python27\Lib\site-packages\w3lib byte-compiling D:\Python27\Lib\site-packages\w3lib\encoding.py to encoding.pyc byte-compiling D:\Python27\Lib\site-packages\w3lib\form.py to form.pyc byte-compiling D:\Python27\Lib\site-packages\w3lib\html.py to html.pyc byte-compiling D:\Python27\Lib\site-packages\w3lib\http.py to http.pyc byte-compiling D:\Python27\Lib\site-packages\w3lib\url.py to url.pyc byte-compiling D:\Python27\Lib\site-packages\w3lib\util.py to util.pyc byte-compiling D:\Python27\Lib\site-packages\w3lib\__init__.py to __init__.pyc running install_egg_info Writing D:\Python27\Lib\site-packages\w3lib-1.2-py2.7.egg-info T:\w3lib-1.2>
验证安装:
T:\>python Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import w3lib >>>
五、 安装libxml2
官方主页:http://users.skynet.be/sbi/libxml-python/http://pypi.python.org/pypi/pyOpenSSL
下载地址:http://users.skynet.be/sbi/libxml-python/binaries/libxml2-python-2.7.7.win32-py2.7.exe
安装过程:略
验证安装:
T:\>python Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import libxml2 >>>
六、 安装pyOpenSSL
官方主页:http://pypi.python.org/pypi/pyOpenSSL
下载地址:http://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.13.winxp32-py2.7.msi
安装过程:略
验证安装:
T:\>python Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import OpenSSL >>>
楼主安装的是python2.7.12版本
以上步骤完成之后直接pip install scrapy 好像并不能用
报错显示
building 'lxml.etree' extension
error: Unable to find vcvarsall.bat
百度了以后说是lxml没装
于是又去安装lxml
lxml下载网站为http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml
进去搜索lxml
在这之前需要先安装wheel,pip install wheel即可
这里要选择对应自己python版本的下载,进入下载目录,pip install 完整文件名
如果版本不对
在安装过程会显示这个报错
lxml安装完成
接下来就可以安装scrapy了
pip install scrapy
安装完成后 输入 scrapy命令 显示如下 就完成了...
我在装完之后写了个小demo
结果运行scrapy crawl dmoz
报如下错误
重点在最后一句话
百度后发现是没用装win32依赖
下载地址在https://sourceforge.net/projects/pywin32/files/pywin32/Build%20220/
要选择符合自己版本的win32下载
下载完成之后运行即可