day14 ~ 增量式与分布式
昨日回顾:
1.redis的安装
1.将安装包解压到一个文件夹下: 如 D:\redis, 会在该文件夹下看到所有redis的文件
2.将该文件夹添加至系统环境变量中
3.在解压的文件目录的地址栏上输入cmd, 在cmd窗口中输入redis-server ./redis.windows.conf , 然后回车, 如果出现下面图片的样子说明redis安装成功了
[外链图片转存失败(img-shSyltG1-1566217990351)(C:\Users\Administrator\AppData\Roaming\Typora\typora-user-images\1565516450167.png)]
2.基于crawlSpider的全站数据爬取
# 项目的创建
scrapy startproject projectname
scrapy genspider -t crawl spidername www.baidu.com
# crawlspider全站数据爬取:
- CrawlSpider是一个爬虫类, 是scrapy.spider的子类, 功能比spider更强大.
- CrawlSpider的机制:
- 连接提取器: 可以根据指定的规则进行连接的提取
- 规则解析器: 更具指定的规则对响应数据进行解析
# 案例: 基于CrawlSpider对笑话网进行全站深度数据爬取, 抓取笑话标题与内容, 并存储于MongoDB中
# item编码:
import scrapy
class JokeItem(scrapy.Item):
title = scrapy.Field()
content = scrapy.Field()
# spider编码:
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from..items import JokeItem
class ZSpider(CrawlSpider):
name = 'z'
# allowed_domains = ['www.baidu.com']
start_urls = ['http://xiaohua.zol.com.cn/lengxiaohua/']
link = LinkExtractor(allow=r'/lengxiaohua/\d+.html')
link_detail = LinkExtractor(allow=r'.*?\d+\.html')
rules = (
Rule(link, callback='parse_item', follow=True),
Rule(link_detail, callback='parse_detail'),
)
def parse_item(self, response):
pass
def parse_detail(self, response):
title = response.xpath('//h1[@class="article-title"]/text()').extract_first()
content = response.xpath('//div[@class="article-text"]//text()').extract()
content = ''.join(content)
if title and content:
item = JokeItem()
item["title"] = title
item["content"] = content
print(dict(item))
yield item
# pipeline编码:
class JokePipeline(object):
def __init__(self, mongo_uri, mongo_db):
self.mongo_uri = mongo_uri
self.mongo_db = mongo_db
@classmethod
def from_crawler(cls, crawler):
return cls(
mongo_uri=crawler.settings.get('MONGO_URI'),
mongo_db=crawler.settings.get('MONGO_DB')
)
def open_spider(self, spider):
self.client = pymongo.MongoClient(self.mongo_uri)
self.db = self.client[self.mongo_db]
def process_item(self, item, spider):
self.db["joke"].insert(dict(item))
return item
def close(self, spider):
self.client.close()
# 电影天堂: 全