mac安装pyspider会遇到的4个大坑

 安装pyspider

pip3 install pyspider

 

安装完成后运行pyspider，会显示async报错（错误一）

~$ pyspider 
Traceback (most recent call last):
  File "/usr/local/bin/pyspider", line 5, in <module>
    from pyspider.run import main
  File "/usr/local/lib/python3.7/site-packages/pyspider/run.py", line 231
    async=True, get_object=False, no_input=False):
        ^
SyntaxError: invalid syntax

因为async为pyton的关键字不能使用，所以需要把async替换为别的 

根据错误信息进入相对应的路径下如：/usr/local/lib/python3.7/site-packages/pyspider

使用ack命令进行查看，如果没有ack可以进行安装：brew install ack

～$ ack async
run.py
231:            async=True, get_object=False, no_input=False):
245:                      poolsize=poolsize, proxy=proxy, async=async)
365:        webui_fetcher = ctx.invoke(fetcher, async=False, get_object=True, no_input=True, **fetcher_config)

fetcher/tornado_fetcher.py
81:    def __init__(self, inqueue, outqueue, poolsize=100, proxy=None, async=True):
89:        self.async = async
95:        if self.async:
117:        if self.async:
118:            return self.async_fetch(task, callback)
120:            return self.async_fetch(task, callback).result()
123:    def async_fetch(self, task, callback=None):
155:            return self.ioloop.run_sync(functools.partial(self.async_fetch, task, lambda t, _, r: True))

webui/app.py
95:    'fetch': lambda x: tornado_fetcher.Fetcher(None, None, async=False).fetch(x),

红色框里面是需要修改的，将async替换为async_

修改完后运行：～$ pyspider  

可能又报错了 （错误二）

$ pyspider
phantomjs fetcher running on port 25555
[I 200209 17:01:34 result_worker:49] result_worker starting...
[I 200209 17:01:34 processor:211] processor starting...
Process Process-4:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pyspider/run.py", line 236, in fetcher