![](https://img-blog.csdnimg.cn/20201014180756738.png?x-oss-process=image/resize,m_fixed,h_64,w_64)
scrapy
穆洛玄
这个作者很懒,什么都没留下…
展开
-
pyhton多线程调用scrapy框架
# -*- coding: utf-8 -*-import threadingimport osfrom time import sleepdef crawl(): os.system('scrapy crawl spider_name -s LOG_FILE=all.log') # 不想看到控制台打印debug信息 就加 -s LOG_FILE=all.log 【将debug信息接入all.log文件】if __name__ == '__main__': wh.原创 2020-12-08 17:05:29 · 195 阅读 · 0 评论 -
使用reactor多线程运行scrapy
# -*- coding: utf-8 -*-import threadingfrom twisted.internet import reactor, deferfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.project import get_project_settingsrunner = CrawlerRunner(get_project_settings())@defer.inlineCallbacksdef.原创 2020-12-03 16:37:11 · 409 阅读 · 0 评论 -
scrapy项目的循环启动(并去掉烦人的日志)
# -*- coding: utf-8 -*-from twisted.internet import reactor, deferfrom scrapy.crawler import CrawlerRunnerfrom scrapy.utils.log import configure_loggingimport timeimport loggingfrom scrapy.utils.project import get_project_settings# 在控制台打印日志con.原创 2020-11-26 11:39:09 · 1030 阅读 · 0 评论