scrapyd部署scrapy项目及定时启动，SpiderKeeper(爬虫监控)

scrapyd部署scrapy项目

安装scrapyd　和scrapyd-client

 

pip install scrapyd　

pip install scrapyd-client

发布项目

首先切换到项目的目录，scrapy.cfg记录了项目的配置信息

 

├── scrapy.cfg #项目配置目录
└── Test
    ├── __init__.py
    ├── __init__.pyc
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    ├── settings.pyc
    └── spiders
        ├── __init__.py
        └── __init__.pyc

scrapy.cfg的内容

 

[settings]
default = Test.settings

[deploy:Myploy]#发布名称
url = http://localhost:6800/　#项目发布到那个地址
project = Test#项目名称

接下来写一个简单的爬虫，爬取scrapy首页标题

 

＃Spd.py 

import scrapy
import logging

class TestSpider(scrapy.Spider):
    name = "spd"#爬虫名称，调度的时候要用到
    logger = logging.getLogger('Mylogger')
    start_urls = ['https://scrapy.org/doc/']

    def parse(self,resp