https://www.cnblogs.com/yunlongaimeng/p/11526466.html
1. 在项目文件夹中新建一个commands文件夹
2. 在command的文件夹中新建一个文件 crawlall.py
3.在crawlall.py 中写一个command类,该类继承 scrapy.commands
from scrapy.commands import ScrapyCommand
class Command(ScrapyCommand):
requires_project = True
def syntax(self):
return '[options]'
def short_desc(self):
return 'Runs all of the spiders 运行多个爬虫文件'
def run(self, args, opts):
spider_list = self.crawler_process.spiders.list()
print('*'*100)
print(spider_list)
print('*'*100)
for name in spider_list:
self.crawler_process.crawl(name, **opts.__dict__)
self.crawler_process.start()
到这里还没完,settings.py配置文件还需要加一条。
COMMANDS_MODULE = ‘项目名称.目录名称’
COMMANDS_MODULE = 'news_spider.commands'
命令行执行:启动所有爬虫 scrapy crawlall