搜索引擎五：Scrapy抓取数据入库

目标

• 抓取相关网页的概要(summary)和链接(url)入库
• Scrapy相关原理参考：Scrapy原理分析

设置item.py文件

# -*- coding: utf-8 -*-
import scrapy

#设置要抓取的字段
#summary和url
class UrlItem(scrapy.Item):

    summary = scrapy.Field()
    url = scrapy.Field()

新建爬虫文件

#-*- coding: utf-8 -*-  

from scrapy.contrib.spiders import CrawlSpider, Rule  
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor  
from url.items import UrlItem  

#爬取相关页面的summary和url
#存进页面的item列表里
class URLSpider(CrawlSpider):  

    name = "url_spider"  

    allowed_domains = ["baidu.com"]  

    start_urls=["http://baike.baidu.com/fenlei/%E7%BE%8E%E9%A3%9F?limit=30&index=1&offset=0#gotoList"]  

    rules = [  
        Rule(SgmlLinkExtractor(allow=('.*#gotoList')),  callback = 'parse_url', follow=True),        
        ]  

    def parse_url(self, response):  
        item = UrlItem()  

        item['url'] = response.selector.xpath('//div[@class="list"]/a/@href').extract()
        item['summary'] = response.selector.xpath('//div[@class="list"]/a/text()').extract()

        yield item  

设定pipelines.py文件处理item信息入库

• 要设置mysql数据库编码为：utf-8

# -*- coding: utf-8 -*-
import os     
import MySQLdb  
from scrapy.exceptions import DropItem
import sys
reload(sys)
sys.setdefaultencoding("utf-8")

INDEX = 1
DATE = "2015-02-26"

class UrlPipeline(object):

    def __init__(self):  
        try:  
            self.db = MySQLdb.connect(host="127.0.0.1", user="root", passwd="1234", port=3306, db="html_url",  charset="utf8")  
            self.cursor = self.db.cursor()  
            print "Connect to db successfully!"  

        except:  
            print "Fail to connect to db!" 

    def process_item(self, item, spider):
        global INDEX
        global DATE
        if item['summary']:
            u = 0
            for summary in item['summary']:
                url = 'http://baike.baidu.com'+item['url'][u]
                param = (INDEX, summary, url, DATE, '1')  
                sql = "insert into documents (id,summary,url,date_info,group_id) values(%s,%s,%s,%s,%s)"  
                self.cursor.execute(sql, param)
                u = u + 1
                INDEX = INDEX + 1  

        else:
            raise DropItem(item) 

        return item  


    def close_spider(self, spider):
        self.db.commit() 
        self.db.close 
        print("Done")  

设置settings.py启动pipeline功能

# -*- coding: utf-8 -*-

BOT_NAME = 'url'

SPIDER_MODULES = ['url.spiders']
NEWSPIDER_MODULE = 'url.spiders'

#启动UrlPipeline的功能
ITEM_PIPELINES={'url.pipelines.UrlPipeline': 1}  

启动爬虫抓取入库

• 项目根目录：scrapy crawl url_spider
• 现在数据库数据：388条菜式和对应url

小结

• 现在已经准备好数据了，下一步编写Django的结果页面用于返回结果
• 索引可以使用CoreSeek的实时索引，但是怕内存不足，所以先把数据存本地