python3.9 安装 pyspider

安装pyspider   

 pip  install pyspider

直接报错
Please specify --curl-dir=/path/to/built/libcurl  

于是从PythonLibs官网 中获取依赖并自行下载到本地 

下载与3.9对应的

执行安装

pip install D:\下载\ad3-2.2.1-cp39-cp39-win32.whl

安装成功之后 执行 pip install pyspider ,提示安装成功,运行 pyspider 报错。

Traceback (most recent call last):
  File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Python39\Scripts\pyspider.exe\__main__.py", line 4, in <module>
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 231
    async=True, get_object=False, no_input=False):
    ^
SyntaxError: invalid syntax

打开run.py  把async 替换成async1

继续执行pyspider ,还是报错

aceback (most recent call last):
  File "D:\Python39\lib\threading.py", line 980, in _bootstrap_inner
    self.run()
  File "D:\Python39\lib\threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 236, in fetcher
    Fetcher = load_cls(None, None, fetcher_cls)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 48, in load_cls
    return utils.load_object(value)
  File "D:\Python39\lib\site-packages\pyspider\libs\utils.py", line 369, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "D:\Python39\lib\site-packages\pyspider\fetcher\__init__.py", line 1, in <module>
    from .tornado_fetcher import Fetcher
  File "D:\Python39\lib\site-packages\pyspider\fetcher\tornado_fetcher.py", line 81
    def __init__(self, inqueue, outqueue, poolsize=100, proxy=None, async=True):

打开  tornado_fetcher.py   把async 替换成async1 ,在执行pyspider,成功了,不再报错如下:

遇到如下错误:


  File "\python39\lib\site-packages
\wsgidav\wsgidav_app.py", line 135, in __init__
    _check_config(config)
  File "python39\lib\site-packages
\wsgidav\wsgidav_app.py", line 119, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'domaincontroller': use 'http_authenticator.domain_control

 解决方案:

找到 文件 找到domaincontroller,这一行按照提示修改,但是依然报错,我就直接删除了,删除之后启动就报如下错误。

TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst

我看到有的帖子说是wsgidav 版本太高,是的就是版本高了,高版本中的方法变了,于是乎复制  getResourceInst  改名为get_resource_inst 增加一下实现

 

 于是重新执行 pyspider all 报错如下:

  File "D:\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "D:\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "D:\Python39\Scripts\pyspider.exe\__main__.py", line 7, in <module>
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 754, in main
    cli()
  File "D:\Python39\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "D:\Python39\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "D:\Python39\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 497, in all
    ctx.invoke(webui, **webui_config)
  File "D:\Python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "D:\Python39\lib\site-packages\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "D:\Python39\lib\site-packages\pyspider\run.py", line 384, in webui
    app.run(host=host, port=port)
  File "D:\Python39\lib\site-packages\pyspider\webui\app.py", line 64, in run
    from werkzeug.dispatcher import DispatcherMiddleware
ModuleNotFoundError: No module named 'werkzeug.dispatcher'

依然是安装了新版本之后依赖包找不到了,于是找到 site-packages\werkzeug 安装目录发现werkzeug\middleware 下有个dispatcher.py 中有想要的类,于是就把webui 的引用包改了一下

 

 

        if dav_app:
            from werkzeug.middleware.dispatcher import DispatcherMiddleware
            application = DispatcherMiddleware(application, {
                '/dav': dav_app
            })

        container = tornado.wsgi.WSGIContainer(application)

于是乎就愉快的跑起来了。

 经历两天奋战终于搞定了。

本次安装思路就是:安装新版本,遇到依赖报错就去找新版本的包中类库所在的路径,修改对应的依赖文件中的依赖。但是安装之后修改过的痕迹被覆盖的话就得重新搞了。这个安装方式适合有一定的开发经验的朋友。

 

 

  • 0
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值