scrapyd部署scrapy项目
安装scrapyd 和scrapyd-client
pip install scrapyd
pip install scrapyd-client
发布项目
首先切换到项目的目录,scrapy.cfg记录了项目的配置信息
├── scrapy.cfg #项目配置目录
└── Test
├── __init__.py
├── __init__.pyc
├── items.py
├── middlewares.py
├── pipelines.py
├── settings.py
├── settings.pyc
└── spiders
├── __init__.py
└── __init__.pyc
scrapy.cfg的内容
[settings]
default = Test.settings
[deploy:Myploy]#发布名称
url = http://localhost:6800/ #项目发布到那个地址
project = Test#项目名称
接下来写一个简单的爬虫,爬取scrapy首页标题
#Spd.py
import scrapy
import logging
class TestSpider(scrapy.Spider):
name = "spd"#爬虫名称,调度的时候要用到
logger = logging.getLogger('Mylogger')
start_urls = ['https://scrapy.org/doc/']
def parse(self,resp