增量式与分布式

day14 ~ 增量式与分布式

昨日回顾:

1.redis的安装

1.将安装包解压到一个文件夹下: 如 D:\redis, 会在该文件夹下看到所有redis的文件
2.将该文件夹添加至系统环境变量中
3.在解压的文件目录的地址栏上输入cmd, 在cmd窗口中输入redis-server ./redis.windows.conf , 然后回车, 如果出现下面图片的样子说明redis安装成功了

[外链图片转存失败(img-shSyltG1-1566217990351)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\1565516450167.png)]

---

2.基于crawlSpider的全站数据爬取

# 项目的创建
scrapy startproject projectname
scrapy genspider -t crawl spidername www.baidu.com

# crawlspider全站数据爬取:
- CrawlSpider是一个爬虫类, 是scrapy.spider的子类, 功能比spider更强大.
- CrawlSpider的机制:
    - 连接提取器: 可以根据指定的规则进行连接的提取
    - 规则解析器: 更具指定的规则对响应数据进行解析

# 案例: 基于CrawlSpider对笑话网进行全站深度数据爬取, 抓取笑话标题与内容, 并存储于MongoDB中

# item编码:
import scrapy
class JokeItem(scrapy.Item):
    title = scrapy.Field()
    content = scrapy.Field()

# spider编码:
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from..items import JokeItem


class ZSpider(CrawlSpider):
    name = 'z'
    # allowed_domains = ['www.baidu.com']
    start_urls = ['http://xiaohua.zol.com.cn/lengxiaohua/']
    link = LinkExtractor(allow=r'/lengxiaohua/\d+.html')
    link_detail = LinkExtractor(allow=r'.*?\d+\.html')
    rules = (
        Rule(link, callback='parse_item', follow=True),
        Rule(link_detail, callback='parse_detail'),
    )

    def parse_item(self, response):
        pass

    def parse_detail(self, response):
        title = response.xpath('//h1[@class="article-title"]/text()').extract_first()
        content = response.xpath('//div[@class="article-text"]//text()').extract()
        content = ''.join(content)

        if title and content:
            item = JokeItem()
            item["title"] = title
            item["content"] = content
            print(dict(item))
            yield item

# pipeline编码:
class JokePipeline(object):

    def __init__(self, mongo_uri, mongo_db):
        self.mongo_uri = mongo_uri
        self.mongo_db = mongo_db

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            mongo_uri=crawler.settings.get('MONGO_URI'),
            mongo_db=crawler.settings.get('MONGO_DB')
        )

    def open_spider(self, spider):
        self.client = pymongo.MongoClient(self.mongo_uri)
        self.db = self.client[self.mongo_db]

    def process_item(self, item, spider):
        self.db["joke"].insert(dict(item))
        return item

    def close(self, spider):
        self.client.close()

---

# 电影天堂: 全