Scrapy顺序执行多个爬虫
使用subprocess按照顺序执行多个爬虫,新建一个start.py文件,输入一下内容,
def crawl_work():
subprocess.Popen('scrapy crawl spider1', shell=True).wait()
subprocess.Popen('scrapy crawl spider2', shell=True).wait()
subprocess.Popen('scrapy crawl spider3', shell=True).wait()
subprocess.Popen('scrapy crawl spider4', shell=True).wait()
if __name__ == '__main__':
crawl_work()
需要注意的是,如果你是用shell文件+crontab定时运行爬虫程序的话,需要在shell
脚本中先定位到scrapy项目中再运行,比如:
cd /home/program/YourProject/;/usr/bin/python ./start.py