scrapy xpath css 经典使用

test() 函数

from scrapy import Selector
doc = “””
… “””
sel = Selector(text=doc, type=”html”)
sel.xpath(‘//li//@href’).extract()
[u’link1.html’, u’link2.html’, u’link3.html’, u’link4.html’, u’link5.html’]
sel.xpath(‘//li[re:test(@class, “item-\d$”)]//@href’).extract()
[u’link1.html’, u’link2.html’, u’link4.html’, u’link5.html’]

---

contains() 与 re() 结合使用

response.xpath(‘//a[contains(@href, “image”)]/text()’).re(r’Name:\s*(.*)’)

---

xpath 与 css 模糊匹配的比较

response.xpath(‘//base/@href’).extract()
[u’http://example.com/‘]

response.css(‘base::attr(href)’).extract()
[u’http://example.com/‘]

response.xpath(‘//a[contains(@href, “image”)]/@href’).extract()
[u’image1.html’,
u’image2.html’,
u’image3.html’,
u’image4.html’,
u’image5.html’]

response.css(‘a[href=image]::attr(href)’).extract()*
[u’image1.html’,
u’image2.html’,
u’image3.html’,
u’image4.html’,
u’image5.html’]

response.xpath(‘//a[contains(@href, “image”)]/img/@src’).extract()
[u’image1_thumb.jpg’,
u’image2_thumb.jpg’,
u’image3_thumb.jpg’,
u’image4_thumb.jpg’,
u’image5_thumb.jpg’]

response.css(‘a[href*=image] img::attr(src)’).extract()
[u’image1_thumb.jpg’,
u’image2_thumb.jpg’,
u’image3_thumb.jpg’,
u’image4_thumb.jpg’,
u’image5_thumb.jpg’]

---

contains() 中的 .

sel.xpath(“//a[contains(., ‘Next Page’)]”).extract()
[u’Click here to go to the Next Page‘]

---

css() 和 xpath() 结合使用

from scrapy import Selector
sel = Selector(text=’

Special date

‘)
sel.css(‘.shout’).xpath(‘./time/@datetime’).extract()
[u’2014-07-23 19:00’]

---

匹配中文
response.xpath(u’//a[text()=”滚动”]/@href’).extract()

---