【Scrapy学习】 scrapyd 文件配置

最新推荐文章于 2024-08-08 18:42:02 发布

weixin_34024034

最新推荐文章于 2024-08-08 18:42:02 发布

阅读量497

点赞数

文章标签： python json

原文链接：https://my.oschina.net/whitejavadog/blog/906655

版权

2019独角兽企业重金招聘Python工程师标准>>>

scrapyd配置文件：如果没有没有配置文件，scrapyd会使用自身的默认值，比如默认每个CPU 最多只执行4个scrapy进程。 CentOS 6.5 64 位 scrapy 1.3.3 scrapyd 1.1.1

如果设置了scrapyd的配置文件： scrapyd会搜索路径：

• /etc/scrapyd/scrapyd.conf (Unix)
• c:\scrapyd\scrapyd.conf (Windows)
• /etc/scrapyd/conf.d/* (in alphabetical order, Unix)
• scrapyd.conf
• ~/.scrapyd.conf (users home directory)

我的配置文件放在etc /scrapyd/scrapyd.conf 下

[scrapyd]
eggs_dir = /usr/scrapyd/eggs
logs_dir = /usr/scrapyd/logs
jobs_to_keep = 100
dbs_dir = /usr/scrapyd/dbs
max_proc = 0
max_proc_per_cpu = 800
finished_to_keep = 100
poll_interval = 5.0
bind_address = 192.168.17.30
http_port = 6800
debug = off
runner = scrapyd.runner
application = scrapyd.app.application
launcher = scrapyd.launcher.Launcher
webroot = scrapyd.website.Root
[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs

其中在打开web界面时，如果长时间没有操作，后台会报出Timing out..

转载于:https://my.oschina.net/whitejavadog/blog/906655