spider爬虫,适合meta传参的爬虫(列表页,详情页都有数据要爬取的时候)
crawlspider爬虫,适合不用meta传参的爬虫
scrapy genspider -t crawl it it.com
# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from Sun.items import SunItem
class DongguanSpider(CrawlSpider):
name = 'dongguan'
# 修改允许的域
allowed_domains = ['sun0769.com']
# 修改起始的url
start_urls = ['http://wz.sun0769.com/index.php/question/questionType?type=4']
rules = (
# 构建列表url的提取规则
Rule(LinkExtractor(allow=r'questionType'), follow=True),
# 构建详情页面url提取规则
# 'html/question/201711/352271.shtml'
Rule(LinkExtractor(allow=r'html/question/\d+/\d+.shtml'), callback='parse_item'),
)
def parse_item(self, response):
# print (response.url,'--------')
# 构建item实例
item = SunItem()
# 抽取数据,将数据存放到item中
item['number'] = response.xpath('/html/body/div[6]/div/div[1]/div[1]/strong/text()').extract()[0].split(':')[-1].strip()
item['title'] = response.xpath('/html/body/div[6]/div/div[1]/div[1]/strong/text()').extract()[0].split(':')[-1].split(' ')[0]
item['link'] = response.url
data = ''.join(response.xpath('//div[@class="c1 text14_2"]/text()|//div[@class="contentext"]/text()').extract())
item['content'] = data.replace('\xa0','')
# print(item)
# 返回数据
yield item
链接提取器的使用
scrapy shell http://hr.tencent.com/position.php
>>> from scrapy.linkextractors import LinkExtractor
>>> le = LinkExtractor(allow=('position_detail.php\?id=\d+&keywords=&tid=0&lid=0'))
或者直接 le = LinkExtractor(allow=('position_detail.php')) 也可以
>>> links=le.extract_links(response)
>>> for link in links:
... print(link)
...
>>> for link in links:
... print(link.url)
...