转载请注明出处:http://blog.csdn.net/l1028386804/article/details/79057362
一、场景描述
数据源准备工作详见博文《Python之——自动上传本地log文件到HDFS(基于Hadoop 2.5.2)》。
统计一个网站的HTTP状态码比例数据,可以帮助我们了解网站的可用度及健康状态,比如我们关注的200、404、5xx状态等。在此示例中,我们利用Mrjob的多步调用形式来实现,除了基本的mapper、reducer方法外,还可以添加自定义处理方法,在steps中添加调用即可。
二、实现MapReduce
【/usr/local/python/source/httpstatus.py】
# -*- coding:UTF-8 -*-
'''
Created on 2018年1月14日
@author: liuyazhuang
'''
from mrjob.job import MRJob
import re
class MRCounter(MRJob):
def mapper(self, key, line):
i = 0
for httpcode in line.split():
#获取日志中HTTP状态码段,作为key
if i == 8 and re.match(r"\d{1,3}", httpcode):
#初始化key:value, value计数为1,方便reducer做累加
yield httpcode, 1
i += 1
def reducer(self, httpcode, occurrences):
#累加操作
yield httpcode, sum(occurrences)
def steps(self):
#在steps方法中添加调用队列
return [self.mr(mapper=self.mapper),
self.mr(reducer=self.reducer)]
if __name__ == '__main__':
MRCounter.run()
三、生成MapReduce任务
python httpstatus.py -r hadoop --jobconf mapreduce.job.priority=VERY_HIGH --jobconf mapreduce.map.tasks=2 --jobconf mapduce.reduce.tasks=1 -o hdfs://liuyazhuang121:9000/output/httpstatus hdfs://liuyazhuang121:9000/user/root/website.com/20180114