windows下python3.7安装pyspider坑记

一、概述

本机已经安装好python3.7，现做测试，需要在开发机上安装pyspider，基于windows下的安装遇到一些坑，特此记录

二、安装步骤

pip install pyspider

此过程一般不会有问题，因为pip会自动安装依赖包，但能不能正常运行起来，它管不了那么多了哦~ 

三、开始填坑

1、执行 pyspider all，立马报错：

Traceback (most recent call last):
  File "D:\Program Files\python37\Scripts\pyspider-script.py", line 11, in <module>
    load_entry_point('pyspider==0.3.10', 'console_scripts', 'pyspider')()
  File "d:\program files\python37\lib\site-packages\pkg_resources\__init__.py", line 487, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "d:\program files\python37\lib\site-packages\pkg_resources\__init__.py", line 2728, in load_entry_point
    return ep.load()
  File "d:\program files\python37\lib\site-packages\pkg_resources\__init__.py", line 2346, in load
    return self.resolve()
  File "d:\program files\python37\lib\site-packages\pkg_resources\__init__.py", line 2352, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "d:\program files\python37\lib\site-packages\pyspider\run.py", line 231
    async=True, get_object=False, no_input=False):
        ^
SyntaxError: invalid syntax

 这是因为python3.5开始把async和await设为关键字了，这里作为参数名与关键字冲突了，因此找到对应的py文件，将async修改为其他名称，我修改为asyn。在以下几个文件中存在：

pyspider->run.py、pyspider->fetcher->tornado_fetcher.py、webui>app.py。

2、再执行，再报错：

  File "d:\program files\python37\lib\site-packages\pyspider\webui\app.py", line 59, in run
    from .webdav import dav_app
  File "d:\program files\python37\lib\site-packages\pyspider\webui\webdav.py", line 216, in <module>
    dav_app = WsgiDAVApp(config)
  File "d:\program files\python37\lib\site-packages\wsgidav\wsgidav_app.py", line 134, in __init__
    _check_config(config)
  File "d:\program files\python37\lib\site-packages\wsgidav\wsgidav_app.py", line 118, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_controller' instead.

这个是因为wsgidav版本太高了，重新安装一下：

python -m pip uninstall wsgidav
python -m pip install wsgidav==2.4.1

 至此，有些小伙伴就应该可以正常启动pyspider了，我的环境还遇到以下错误：

3、再报错：

  File "d:\program files\python37\lib\site-packages\pyspider\run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "d:\program files\python37\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "d:\program files\python37\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "d:\program files\python37\lib\site-packages\pyspider\run.py", line 384, in webui
    app.run(host=host, port=port)
  File "d:\program files\python37\lib\site-packages\pyspider\webui\app.py", line 64, in run
    from werkzeug.wsgi import DispatcherMiddleware
ImportError: cannot import name 'DispatcherMiddleware' from 'werkzeug.wsgi' (d:\program files\python37\lib\site-packages\werkzeug\wsgi.py)

经过多次尝试，发现是 werkzeug版本高了，DispatcherMiddleware这个类已经不存在了，因此重新安装小版本：

python -m pip uninstall werkzeug
python -m pip install werkzeug==0.15

为什么是0.15版本？我是再卸载后，暂不安装，直接执行pyspider时，错误提示信息中显示了最低版本为0.15：）

4、正常运行，截图纪念

---

 补充：

在不同的PC上安装后，相关依赖项版本不同，还可能出现不同的问题：

• 启动时卡在result_worker starting...
  可带参数逐一启动，排除问题：
  • pyspider webui，单独启动webui
  • pyspider result_worker，单独启动result_worker
  • 等等，查看是否报错
• 启动报cannot import name 'DispatcherMiddleware'
  可修改pyspider/webui/app.py文件的第64行：
  将from werzeug.wsgi import DispatcherMiddleware
  改为from werkzeug.middleware.dispatcher import DispatcherMiddleware