python-scrapy-scrapyd-geapy

学习自：scrapyd使用修改api
For more information about the API, see the Scrapyd documentation

相关包安装

略

启动服务

1. 修改 ~/.env/env_name/lib/python3.6/site-packages/scrapyd/default_scrapyd.conf ==> bind_address = 0.0.0.0
2. 启动服务: scrapyd

web访问服务

xxx.xx.xx.xxx:6800

修改 ~/.env/env_name/lib/python3.6/site-packages/website.py ==> insert


<p><code>curl http://localhost:6800/schedule.json -d project=default -d spider=somespider</code></p>
<p><code> curl http://localhost:6800/cancel.json -d project=myproject -d job=6487ec79947edab326d6db28a2d86511e8247444 </code></p>
<p><code> curl http://localhost:6800/listprojects.json </code></p>
<p><code> curl http://localhost:6800/listversions.json?project=myproject </code></p>
<p><code> curl http://localhost:6800/listspiders.json?project=myproject </code></p>
<p><code>  curl http://localhost:6800/listjobs.json?project=myproject </code></p>
<p><code> curl http://localhost:6800/delproject.json -d project=myproject </code></p>
<p><code> curl http://localhost:6800/delversion.json -d project=myproject -d version=r99 </code></p>

至此，scrapyd服务启动完成

通过一个实际scrapy例子演示

修改默认scrapy的配置文件

1. 加个自定义的名字
2. url 修改为scrapyd服务一致的ip和端口

cd scrapy_demo

vim scrapy.cfg ==>

[deploy:scrapy_demo]
# url = http://localhost:6800/
url = http://xxx.xx.xx.xxx:6800/
project = project_name

# cd 到 scrapy.cfg 同级目录下 运行命令

scrapyd-deploy -l        #

win中scrapyd-deploy不能运行的话

在python运行环境中新建scrapyd-deploy.bat文件
在：D:\Envs\env_name\Scripts\scrapyd-deploy.bat

@echo off
"D:\Envs\env_name\Scripts\python.exe" "D:\Envs\env_name\Scripts\scrapyd-deploy" %*

然后运行scrapyd-deploy -l即可执行成功！

curl+json格式 操作 爬虫脚本

1. list project

curl http://localhost:6800/listprojects.json

2. list spiders

curl http://localhost:6800/listspiders.json?project=project_name

3. run crawl

curl http://localhost:6800/schedule.json -d project=project_name -d spider=spider_name

4. cancel crawl

curl http://localhost:6800/cancel.json -d project=project_name -d job=2157910a9ef811e995c020040fe78714

5. delete project

curl http://localhost:6800/delproject.json -d project=project_name 

5. delete version = r99

curl http://localhost:6800/delversion.json -d project=project_name -d version=r99

gerapy-使用

源码获取

github地址：https://github.com/Gerapy/Gerapy
自己修改、提出关键服务，修改task/status页面的错误yyyy-MM-dd HH:mm:ss，私有仓库

启动服务

# back_end: django server
cd django_server
python manage.py migrate
python manage.py createsuperuser
python manage.py runserver 0.0.0.0:8000

# front_end: vue server
cd vue_server
# modify proxy targe port ==> 8000
npm run serve[这个根据自己的设置]

view web_page

localhost:8080