[python小记]scrapy-pipeline存储item到excel中-openpyxl

首先,话不多说,先上scrapy-item pipeline

之前数据都是存到数据库和json中,用的时候还得转成excel格式,挺麻烦,所以今天查了一下发现了openpyxl这个库,在此小记..

from openpyx import WorkBook
#创建工作簿,同时页建一个sheet
wb = WorkBook()
#调用得到的sheet,并命名为test1
ws = wb.active 
(注:active返回的是一个列表) 
@property
    def active(self):
        """Get the currently active sheet or None
        
        :type: :class:`openpyxl.worksheet.worksheet.Worksheet`
        """
        try:
            return self._sheets[self._active_sheet_index]
        except IndexError:
            pass 
ws.title = 'test1'
#插入数据
ws.append([...])
#保存工作簿,在当前目录下文件名为test1.xlsx
wb.save('test1.xlsx')

项目代码

class CdcspiderExcelPipeline(object):
    '''
    use Item Exporter
    save the item to excel
    '''

    def __init__(self):
        '''
        initialize the object
        '''
        self.spider = None
        self.count = 0

    def log(self, l):
        '''
        reload the log
        :return:
        '''
        msg = '========== CdcspiderExcelPipeline ==  %s' % l

        if self.spider is not None:
            # spider.logger -> return logging.LoggerAdapter(logger, {'spider': self})
            self.spider.logger.info(msg)

    def open_spider(self, spider):
        '''
        create a queue
        :return:
        '''
        self.wb = openpyxl.Workbook()
        self.ws = self.wb.active
        self.ws.append(['文章日期', '文章标题', 'url', '文章作者'])

    def process_item(self, item, spider):
        '''
        save every
        :return:
        '''
        self.log('process %s, %s:' % (spider.name, self.count + 1))

        line = [item['article_time'],item['title'],item['url'],item['author']]
        self.ws.append(line)
        return item

    def close_spider(self, spider):
        '''
        save lines to excel
        :return:
        '''
        print 'ExcelPipline info:  items size: %s' % self.count
        file_name = _generate_filename(spider, file_format='xlsx')
        self.wb.save(file_name)

结果如下

此外,scrapy提供了item exporter进行持久化或者导出,但笔者本人使用觉着不如第三方库方便,当然可能跟小编水平有限相关哈哈.