还是用xml的模板爬xml 来的直接。
TestItem in items.py:
class TestItem(scrapy.Item):
id = scrapy.Field()
name = scrapy.Field()
description = scrapy.Field()
main:
from scrapy import log
from scrapy.contrib.spiders import XMLFeedSpider
from myproject.items import TestItem
class MySpider(XMLFeedSpider):
name = 'example'
start_urls = ['portfolio xii xml']
iterator = 'xml' # This is actually unnecessary, since it's the default value
itertag = 'PortfolioStatistics' #Name of Parent Node
def parse_node(self, response, node):
item = TestItem()
item['id'] = node.xpath('@_SalePosition').extract() #Get Parent Node Attribute
item['name'] = node.xpath('EquityStyle/EquityStyleCoreValue/StyleBox/text()').extract() #Get son of grandson
item['description'] = node.xpath('BondStatistics/EffectiveDuration/text()').extract() #Get Grandson
return item
Output:
[
{“description”: [“0.53”], “name”: [“6”], “id”: [“L”]}
]
再进阶觉得这个例子不错,特别是pipeline那部分:
http://www.jianshu.com/p/e71ff6173302
2017-04-02:后续去抓带basic authentication的案例,会需要重写start_request才行。