![](https://img-blog.csdnimg.cn/20201014180756738.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
scrapy
穆洛玄
这个作者很懒,什么都没留下…
展开
-
pyhton多线程调用scrapy框架
# -*- coding: utf-8 -*- import threading import os from time import sleep def crawl(): os.system('scrapy crawl spider_name -s LOG_FILE=all.log') # 不想看到控制台打印debug信息 就加 -s LOG_FILE=all.log 【将debug信息接入all.log文件】 if __name__ == '__main__': wh.原创 2020-12-08 17:05:29 · 192 阅读 · 0 评论 -
使用reactor多线程运行scrapy
# -*- coding: utf-8 -*- import threading from twisted.internet import reactor, defer from scrapy.crawler import CrawlerRunner from scrapy.utils.project import get_project_settings runner = CrawlerRunner(get_project_settings()) @defer.inlineCallbacks def.原创 2020-12-03 16:37:11 · 408 阅读 · 0 评论 -
scrapy项目的循环启动(并去掉烦人的日志)
# -*- coding: utf-8 -*- from twisted.internet import reactor, defer from scrapy.crawler import CrawlerRunner from scrapy.utils.log import configure_logging import time import logging from scrapy.utils.project import get_project_settings # 在控制台打印日志 con.原创 2020-11-26 11:39:09 · 1008 阅读 · 0 评论