scrapy中spider

1抓取单个网页：
parse(response)：回调函数中response表示url中提取的内容

2抓取多个网页(并且链接其他网页)
方法1：

  rules = (
       ##下面是符合规则的网址,但是不抓取内容,只是提取该页的链接
       #Rule(SgmlLinkExtractor(allow=('huhuuu/default.html?page=([w]+)',), )),
                 Rule(LinkExtractor(allow('A_Eagle/article/list/(\d+)',)),),
       #Rule(LinkExtractor(allow='learning/index&page=([d]+)')),
       ##下面是符合规则的网址,提取内容
 Rule(LinkExtractor(allow=('a_eagle/article/details/(\d+)',)), callback='parse_item',),
    )   

parse_item(response)：回调函数中response表示第二个Rule中提取的内容
解析
此时的response为

a_eagle/article/details/(\d+)

中的内容，

sel = Selector(response)
items[‘title’] = sel.xpath(‘//div[@id=”article_details”]/div[@class=”article_title”]/h1/span/a/text()’).extract()
item[‘url’]=response
yield item

**方法2：**
**以上两个也可以合并为：**

rules = (
Rule(LinkExtractor(allow=’learning/detail/(\d+)’), callback=’parse_item’, ),follow=True
)
)

sel = Selector(response)
items[‘title’]=sel.xpath(‘//div[@class=”article_title”]/h1/span/a/text()’).extract()
items[‘url’]=sel.xpath(‘//div[@class=”article_title”]/h1/span/a/@href’).extract()

此时的response为

learning/detail/(\d+)

中的内容

3
Rule中：如果没有回调函数，follow默认为True，表示跟进；
             如果有回调函数，则follow默认为False,表示不跟进
同时：xpath()返回的是list()
text()得出的文本就是标题，@herf得到的就是链接，extract以unicode编码返回。 
**例子**：以抓取朋友的博客为例http://blog.csdn.net/A_Eagle
**方法1：**

class CsdnSpider(CrawlSpider):
name = ‘csdn’
allowed_domains = [‘blog.csdn.net’]
start_urls = [‘http://blog.csdn.net/A_Eagle‘]

rules = (
    Rule(LinkExtractor(allow=('A_Eagle/article/list/(\d+)',)),),
    Rule(LinkExtractor(allow=('a_eagle/article/details/(\d+)',)), callback='parse_item',),
)

def parse_item(self, response):
sel = Selector(response)
items=CsdnBoKeItem()
items[‘title’] = sel.xpath(‘//div[@id=”article_details”]/div[@class=”article_title”]/h1/span/a/text()’).extract()
items[‘url’] =response
print items
yield items

**方法2：**

class CsdnSpider(CrawlSpider):
name = ‘csdn’
allowed_domains = [‘blog.csdn.net’]
start_urls = [‘http://blog.csdn.net/A_Eagle‘]
rules = (
Rule(LinkExtractor(allow=(‘A_Eagle/article/list/(\d+)’,)), callback=’parse_item’, follow=True),
)
def parse_item(self, response):
response.xpath(‘//div[@id=”description”]’).extract()
sel = Selector(response)
title= sel.xpath(‘//div[@class=”article_title”]/h1/span/a/text()’).extract()
urls=sel.xpath(‘//div[@class=”article_title”]/h1/span/a/@href’).extract()
for i,j in zip(title,urls):
items = CsdnBoKeItem()
items[‘title’]=i
items[‘url’]=j
print items
yield items
“`

参考：http://blog.csdn.net/zhx6044/article/details/45698535
http://blog.csdn.net/zhx6044/article/details/45698535
http://blog.csdn.net/qy20115549/article/details/52575291