近日,有人和我说分析log日志。
之前,就写过,但是忘了总结了,找了半天也没有找到,看了以后要将东西整理了。
无奈,在网上收拾,看到这个人写的,索性,就搬过来,待我找到我写的,在一块补充一下!
所有网站的服务器上都会保留访问的log日志。这些log日志记录的其他机器访问服务器的ip,时间,http协议,状态码等信息。
比如这样:
![](https://i-blog.csdnimg.cn/blog_migrate/a348021fe78a3e2403c83fd01f4593b2.png)
image.png
大型网站的服务器往往会产生海量的log日志,用hadoop来分析log日志,也是一个很好的练手的机会。
下面写一个例子,通过分析服务器的log日志,统计访问服务器的ip地址和访问的次数。
map函数
public class worldcount extends Mapper<LongWritable,Text,Text,IntWritable>{ //重载Mapper类的map方法 // 这里的key是读取文件的行号,value是对应行号的文本 protected void map(LongWritable key,Text value,Context context) throws IOException, InterruptedException{ //将这一行转化为string String line=value.toString(); //以空格切分 String [] linewords = line.split(" "); //获得ip String ip=linewords[0]; // 所以在context里面写的内容就是 key:ip ,value 是1 context.write(new Text(ip), new IntWritable(1)); } }
<br />
reduce 函数
public class worldcountreduce extends Reducer <Text,IntWritable,Text,IntWritable> { // 一组相同的key,调用一次reduce //相当于调用一次 ,计算一个key对应的个数 protected void reduce (Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException{ //统计单词数 int count=0; for(IntWritable value :values){ count=count+value.get(); } //将输出的结果放到context 里面 context.write(key,new IntWritable(count)); } }
<br />
Main 函数
public class jobclient { public static void main(String []args) throws IOException, ReflectiveOperationException, InterruptedException{ Configuration conf=new Configuration(); //conf.set("yarn.resoucemanager.hostname", value); Job job=Job.getInstance(conf); //job.setJar("~/code/WordCount.jar"); //告知客户端的提交器 mr程序所在的jar包 //这样就不必使用setjar 这样的方法了 job.setJarByClass(jobclient.class); // 告知mrapp master ,map 和reduce 对应的实现类 job.setMapperClass(worldcount.class); job.setReducerClass(worldcountreduce.class); //告知输入,和输出的数据结构的类型 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); //告知mrappmaster 我们启动的reduce tash的数量 //启动maptask 的数量 是yarn 会自动的计算 job.setNumReduceTasks(3); //指定一个目录而不是文件 FileInputFormat.setInputPaths(job, new Path("hdfs://localhost:9000/kpi/")); FileOutputFormat.setOutputPath(job,new Path("hdfs://localhost:9000/kpi/output/")); // job.submit() //这个要比job.submit 要好,因为这个client并不会在提交任务之后,就退出,而是创建一个线程去监控 map和reduce的运行 boolean res=job.waitForCompletion(true); // 执行成功 状态吗 是0,执行失败 状态码是100 // 通过echo $? 显示状态码 System.out.println("wakakka "); System.exit(res?0:100); } }
这里的输入是一个目录,可以把输入的文件放到这个目录里面就好。比如这里,我把access.log.10文件放在kpi目录下面。
![](https://i-blog.csdnimg.cn/blog_migrate/0bededd49a1edfc0015244d815e1f1a1.png)
2017-03-24 22-56-13屏幕截图.png
然后将代码打包为一个jar包,使用hadoo命令执行这个jar包。(执行这条命令必须在jar包所在的目录下面执行)
![](https://i-blog.csdnimg.cn/blog_migrate/4b6bdef81a2eb076c616a8fc3737c1c6.png)
2017-03-24 22-57-33屏幕截图.png
程序运行的结果是在输出的目录里面:
![](https://i-blog.csdnimg.cn/blog_migrate/db8484c1f0e030f06ae8ef21b67e6838.png)
2017-03-25 09-38-29屏幕截图.png
查看一个文件,就是程序的运行结果:
![](https://i-blog.csdnimg.cn/blog_migrate/c0144c68466ed65128f75ec9417de884.png)
2017-03-25 09-41-36屏幕截图.png
关于代码的一些小结:
- hadoop经常启动失败,或者出现访问失败的情况。
- 在写代码之前,一定要把所有的关于hadoop,mapreduce的包导入。
- 将代码打包为一个jar包。
github地址:
https://github.com/zhaozhengcoder/hadoop/tree/master/mapreduce_kpi
作者:sexycoder
链接:https://www.jianshu.com/p/f0b7e273539d
放一段事例,待用:
194.237.142.21 - - [18/Sep/2013:06:49:18 +0000] "GET /wp-content/uploads/2013/07/rstudio-git3.png HTTP/1.1" 304 0 "-" "Mozilla/4.0 (compatible;)" 183.49.46.228 - - [18/Sep/2013:06:49:23 +0000] "-" 400 0 "-" "-" 163.177.71.12 - - [18/Sep/2013:06:49:33 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0" 163.177.71.12 - - [18/Sep/2013:06:49:36 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0" 101.226.68.137 - - [18/Sep/2013:06:49:42 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0" 101.226.68.137 - - [18/Sep/2013:06:49:45 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0" 60.208.6.156 - - [18/Sep/2013:06:49:48 +0000] "GET /wp-content/uploads/2013/07/rcassandra.png HTTP/1.0" 200 185524 "http://cos.name/category/software/packages/" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36" 222.68.172.190 - - [18/Sep/2013:06:49:57 +0000] "GET /images/my.jpg HTTP/1.1" 200 19939 "http://www.angularjs.cn/A00n" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36" 222.68.172.190 - - [18/Sep/2013:06:50:08 +0000] "-" 400 0 "-" "-" 183.195.232.138 - - [18/Sep/2013:06:50:16 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0" 183.195.232.138 - - [18/Sep/2013:06:50:16 +0000] "HEAD / HTTP/1.1" 200 20 "-" "DNSPod-Monitor/1.0" 66.249.66.84 - - [18/Sep/2013:06:50:28 +0000] "GET /page/6/ HTTP/1.1" 200 27777 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 221.130.41.168 - - [18/Sep/2013:06:50:37 +0000] "GET /feed/ HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36" 157.55.35.40 - - [18/Sep/2013:06:51:13 +0000] "GET /robots.txt HTTP/1.1" 200 150 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 50.116.27.194 - - [18/Sep/2013:06:51:35 +0000] "POST /wp-cron.php?doing_wp_cron=1379487095.2510800361633300781250 HTTP/1.0" 200 0 "-" "WordPress/3.6; http://blog.fens.me" 58.215.204.118 - - [18/Sep/2013:06:51:35 +0000] "GET /nodejs-socketio-chat/ HTTP/1.1" 200 10818 "http://www.google.com/url?sa=t&rct=j&q=nodejs%20%E5%BC%82%E6%AD%A5%E5%B9%BF%E6%92%AD&source=web&cd=1&cad=rja&ved=0CCgQFjAA&url=%68%74%74%70%3a%2f%2f%62%6c%6f%67%2e%66%65%6e%73%2e%6d%65%2f%6e%6f%64%65%6a%73%2d%73%6f%63%6b%65%74%69%6f%2d%63%68%61%74%2f&ei=rko5UrylAefOiAe7_IGQBw&usg=AFQjCNG6YWoZsJ_bSj8kTnMHcH51hYQkAA&bvm=bv.52288139,d.aGc" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] "GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1" 304 0 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.215.204.118 - - [18/Sep/2013:06:51:35 +0000] "GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1" 304 0 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] "GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1" 304 0 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] "GET /wp-content/uploads/2013/08/chat.png HTTP/1.1" 200 48968 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] "GET /wp-content/uploads/2013/08/chat2.png HTTP/1.1" 200 59852 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.215.204.118 - - [18/Sep/2013:06:51:37 +0000] "GET /wp-content/uploads/2013/08/socketio.png HTTP/1.1" 200 80493 "http://blog.fens.me/nodejs-socketio-chat/" "Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0" 58.248.178.212 - - [18/Sep/2013:06:51:37 +0000] "GET /nodejs-grunt-intro/ HTTP/1.1" 200 51770 "http://blog.fens.me/series-nodejs/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)" 58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] "GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1" 200 7200 "http://blog.fens.me/nodejs-grunt-intro/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)" 58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] "GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1" 200 786 "http://blog.fens.me/nodejs-grunt-intro/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)" 58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] "GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1" 200 45307 "http://blog.fens.me/nodejs-grunt-intro/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)" 58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] "GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1" 200 93128 "http://blog.fens.me/nodejs-grunt-intro/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)" 58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] "GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1" 200 786 "http://blog.fens.me/nodejs-grunt-intro/" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)"