1. 有些scrapy命令,只有在scrapy project根目录下才available,比如crawl命令
2 . scrapy genspider taobao http://detail.tmall.com/item.htm?id=12577759834
自动在spider目录下生成taobao.py
# -*- coding: utf-8 -*-
import scrapy
class TaobaoSpider(scrapy.Spider):
name = "taobao"
allowed_domains = ["http://detail.tmall.com/item.htm?id=12577759834"]
start_urls = (
'http://www.http://detail.tmall.com/item.htm?id=12577759834/',
)
def parse(self, response):
pass
还有其它模板可以用
scrapy genspider taobao2 http://detail.tmall.com/item.htm?id=12577759834 --template=crawl
# -*- coding: utf-8 -*-
import scrapy
from scrapy.contrib.linkextractors import LinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
from project004.items import Project004Item
class Taobao2Spider(CrawlSpider):
name = 'taobao2'
allowed_domains = ['http://detail.tmall.com/item.htm?id=12577759834']
start_urls = ['http://www.http://detail.tmall.com/item.htm?id=12577759834/']
rules = (
Rule(LinkExtractor(allow=r'Items/'), callback='parse_item', follow=True),
)
def parse_item(self, response):
i = Project004Item()
#i['domain_id'] = response.xpath('//input[@id="sid"]/@value').extract()
#i['name'] = response.xpath('//div[@id="name"]').extract()
#i['description'] = response.xpath('//div[@id="description"]').extract()
return i
3.列出当前项目所有spider: scrapy list
4.view命令在浏览器中查看网页内容
scrapy view http://www.example.com/some/page.html
5.查看设置
scrapy settings --get BOT_NAME
6.运行自包含的spider,不需要创建项目
scrapy runspider <spider_file.py>
7.scrapy project的部署: scrapy deploy
部署spider首先要有spider的server环境,一般使用scrapyd
安装scrapyd:pip install scrapyd
文档:http://scrapyd.readthedocs.org/en/latest/install.html
8.所有可用命令
C:\Users\IBM_ADMIN\PycharmProjects\pycrawl\project004>scrapy
Scrapy 0.24.4 - project: project004
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
check Check spider contracts
crawl Run a spider
deploy Deploy project in Scrapyd target
edit Edit spider
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
list List available spiders
parse Parse URL (using its spider) and print the results
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
转载于:https://blog.51cto.com/dingbo/1600296