MapReduce【数据清洗】

已于 2023-05-29 19:16:27 修改
阅读量1.5k 收藏 6
点赞数 1
分类专栏： Hadoop 文章标签： mapreduce 大数据 hadoop java
于 2023-05-29 19:09:07 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/m0_64261982/article/details/130928775
版权

ETL

在运行核心业务MapReduce程序时，往往要进行数据清洗，清理掉不符合要求的数据。清理的过程往往只需要运行Mapper程序，不需要运行Reducer程序。

需求

去除日志中字段个数<=11的日志（用空格分隔字段）

输入数据

194.237.142.21 - - [18/Sep/2013:06:49:18 +0000] “GET /wp-content/uploads/2013/07/rstudio-git3.png HTTP/1.1” 304 0 “-” “Mozilla/4.0 (compatible;)”
183.49.46.228 - - [18/Sep/2013:06:49:23 +0000] “-” 400 0 “-” “-”
163.177.71.12 - - [18/Sep/2013:06:49:33 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
163.177.71.12 - - [18/Sep/2013:06:49:36 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
101.226.68.137 - - [18/Sep/2013:06:49:42 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
101.226.68.137 - - [18/Sep/2013:06:49:45 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
60.208.6.156 - - [18/Sep/2013:06:49:48 +0000] “GET /wp-content/uploads/2013/07/rcassandra.png HTTP/1.0” 200 185524 “http://cos.name/category/software/packages/” “Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36”
222.68.172.190 - - [18/Sep/2013:06:49:57 +0000] “GET /images/my.jpg HTTP/1.1” 200 19939 “http://www.angularjs.cn/A00n” “Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36”
222.68.172.190 - - [18/Sep/2013:06:50:08 +0000] “-” 400 0 “-” “-”
183.195.232.138 - - [18/Sep/2013:06:50:16 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
183.195.232.138 - - [18/Sep/2013:06:50:16 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
66.249.66.84 - - [18/Sep/2013:06:50:28 +0000] “GET /page/6/ HTTP/1.1” 200 27777 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
221.130.41.168 - - [18/Sep/2013:06:50:37 +0000] “GET /feed/ HTTP/1.1” 304 0 “-” “Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36”
157.55.35.40 - - [18/Sep/2013:06:51:13 +0000] “GET /robots.txt HTTP/1.1” 200 150 “-” “Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)”
50.116.27.194 - - [18/Sep/2013:06:51:35 +0000] “POST /wp-cron.php?doing_wp_cron=1379487095.2510800361633300781250 HTTP/1.0” 200 0 “-” “WordPress/3.6; http://blog.fens.me”
58.215.204.118 - - [18/Sep/2013:06:51:35 +0000] “GET /nodejs-socketio-chat/ HTTP/1.1” 200 10818 “http://www.google.com/url?sa=t&rct=j&q=nodejs%20%E5%BC%82%E6%AD%A5%E5%B9%BF%E6%92%AD&source=web&cd=1&cad=rja&ved=0CCgQFjAA&url=%68%74%74%70%3a%2f%2f%62%6c%6f%67%2e%66%65%6e%73%2e%6d%65%2f%6e%6f%64%65%6a%73%2d%73%6f%63%6b%65%74%69%6f%2d%63%68%61%74%2f&ei=rko5UrylAefOiAe7_IGQBw&usg=AFQjCNG6YWoZsJ_bSj8kTnMHcH51hYQkAA&bvm=bv.52288139,d.aGc” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] “GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1” 304 0 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:51:35 +0000] “GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1” 304 0 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] “GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1” 304 0 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] “GET /wp-content/uploads/2013/08/chat.png HTTP/1.1” 200 48968 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:51:36 +0000] “GET /wp-content/uploads/2013/08/chat2.png HTTP/1.1” 200 59852 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:51:37 +0000] “GET /wp-content/uploads/2013/08/socketio.png HTTP/1.1” 200 80493 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.248.178.212 - - [18/Sep/2013:06:51:37 +0000] “GET /nodejs-grunt-intro/ HTTP/1.1” 200 51770 “http://blog.fens.me/series-nodejs/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] “GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1” 200 7200 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] “GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1” 200 786 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] “GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1” 200 45307 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] “GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1” 200 93128 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.248.178.212 - - [18/Sep/2013:06:51:40 +0000] “GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1” 200 786 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.215.204.118 - - [18/Sep/2013:06:51:41 +0000] “-” 400 0 “-” “-”
58.215.204.118 - - [18/Sep/2013:06:51:41 +0000] “-” 400 0 “-” “-”
58.215.204.118 - - [18/Sep/2013:06:51:41 +0000] “-” 400 0 “-” “-”
157.55.35.40 - - [18/Sep/2013:06:51:43 +0000] “GET /rhadoop-java-basic/ HTTP/1.1” 200 26780 “-” “Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)”
58.248.178.212 - - [18/Sep/2013:06:51:43 +0000] “GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1” 200 7200 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.248.178.212 - - [18/Sep/2013:06:51:45 +0000] “GET /wp-content/uploads/2013/08/grunt.png HTTP/1.1” 200 199040 “http://blog.fens.me/nodejs-grunt-intro/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MDDR; InfoPath.2; .NET4.0C)”
58.215.204.118 - - [18/Sep/2013:06:52:26 +0000] “GET /nodejs-async/ HTTP/1.1” 200 12647 “http://blog.fens.me/nodejs-socketio-chat/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:52:27 +0000] “GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1” 304 0 “http://blog.fens.me/nodejs-async/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:52:27 +0000] “GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1” 304 0 “http://blog.fens.me/nodejs-async/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
58.215.204.118 - - [18/Sep/2013:06:52:27 +0000] “GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1” 304 0 “http://blog.fens.me/nodejs-async/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
163.177.71.12 - - [18/Sep/2013:06:52:29 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
58.215.204.118 - - [18/Sep/2013:06:52:29 +0000] “GET /nodejs-async/?cf_action=sync_comments&post_id=2357 HTTP/1.1” 200 48 “http://blog.fens.me/nodejs-async/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
163.177.71.12 - - [18/Sep/2013:06:52:32 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
58.215.204.118 - - [18/Sep/2013:06:52:32 +0000] “-” 400 0 “-” “-”
58.215.204.118 - - [18/Sep/2013:06:52:33 +0000] “-” 400 0 “-” “-”
58.215.204.118 - - [18/Sep/2013:06:52:33 +0000] “-” 400 0 “-” “-”
101.226.68.137 - - [18/Sep/2013:06:52:36 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
101.226.68.137 - - [18/Sep/2013:06:52:39 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
183.195.232.138 - - [18/Sep/2013:06:53:12 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
183.195.232.138 - - [18/Sep/2013:06:53:12 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
222.66.59.174 - - [18/Sep/2013:06:53:30 +0000] “GET /images/my.jpg HTTP/1.1” 200 19939 “http://www.angularjs.cn/A00n” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0”
222.66.59.174 - - [18/Sep/2013:06:53:36 +0000] “-” 400 0 “-” “-”
216.24.201.254 - - [18/Sep/2013:06:53:57 +0000] “GET /feed/ HTTP/1.1” 200 33867 “-” “Mozilla/5.0 (Ubuntu; X11; Linux x86_64; rv:8.0) Gecko/20100101 Firefox/8.0”
58.215.204.118 - - [18/Sep/2013:06:54:48 +0000] “GET /r-json-rjson/ HTTP/1.1” 200 11500 “http://blog.fens.me/nodejs-async/” “Mozilla/5.0 (Windows NT 5.1; rv:23.0) Gecko/20100101 Firefox/23.0”
61.135.216.105 - - [18/Sep/2013:06:54:51 +0000] “GET /comments/feed/ HTTP/1.1” 304 0 “-” “Mozilla/5.0 (compatible;YoudaoFeedFetcher/1.0;http://www.youdao.com/help/reader/faq/topic006/;1 subscribers;)”
222.66.59.174 - - [18/Sep/2013:06:55:19 +0000] “-” 400 0 “-” “-”
163.177.71.12 - - [18/Sep/2013:06:55:24 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
163.177.71.12 - - [18/Sep/2013:06:55:27 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
101.226.68.137 - - [18/Sep/2013:06:55:30 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
101.226.68.137 - - [18/Sep/2013:06:55:33 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
222.66.59.174 - - [18/Sep/2013:06:55:51 +0000] “GET /images/my.jpg HTTP/1.1” 200 19939 “http://www.angularjs.cn/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E)”
183.195.232.138 - - [18/Sep/2013:06:56:07 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
183.195.232.138 - - [18/Sep/2013:06:56:07 +0000] “HEAD / HTTP/1.1” 200 20 “-” “DNSPod-Monitor/1.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /vps-ip-dns/ HTTP/1.1” 200 11403 “http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=6&cad=rja&ved=0CHIQFjAF&url=http%3A%2F%2Fblog.fens.me%2Fvps-ip-dns%2F&ei=j045UrP5AYX22AXsg4G4DQ&usg=AFQjCNGsJfLMNZnwWXNpTSUl6SOEzfF6tg&sig2=YY1oxEybUL7wx3IrVIMfHA&bvm=bv.52288139,d.b2I” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/style.css HTTP/1.1” 200 7554 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1” 200 32851 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1” 200 7200 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1” 200 786 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/js/jquery.cycle.all.min.js HTTP/1.1” 200 7784 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/js/load.js HTTP/1.1” 200 715 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/functions/css/shortcodes.css HTTP/1.1” 200 2899 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/functions/js/shortcode.js HTTP/1.1” 200 333 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/uploads/2013/06/%E5%8A%A8%E6%80%81IP%E8%A7%A3%E6%9E%90.jpg HTTP/1.1” 200 36779 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/uploads/2013/06/wtmart.png HTTP/1.1” 200 26105 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/uploads/2013/06/linux-dns.png HTTP/1.1” 200 14841 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/uploads/2013/06/win-dns.png HTTP/1.1” 200 30694 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/uploads/2013/06/win-ssh.png HTTP/1.1” 200 17524 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-includes/images/smilies/icon_smile.gif HTTP/1.1” 200 174 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/images/natty-logo.png HTTP/1.1” 200 1438 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/uploads/2013/05/favicon.ico HTTP/1.1” 200 1150 “-” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /js/baidu.js HTTP/1.1” 200 249 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /js/google.js HTTP/1.1” 200 475 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/images/slide-bg.png HTTP/1.1” 200 934 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/images/sprites/post-type.png HTTP/1.1” 200 2009 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:41 +0000] “GET /wp-content/themes/silesia/images/ico-twitter.png HTTP/1.1” 200 2128 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:41 +0000] “GET /wp-content/themes/silesia/images/ico-meta.gif HTTP/1.1” 200 73 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/images/crubms-div.png HTTP/1.1” 200 1255 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/images/bullets/5.gif HTTP/1.1” 200 62 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:40 +0000] “GET /wp-content/themes/silesia/images/home-ico.png HTTP/1.1” 200 1103 “http://blog.fens.me/wp-content/themes/silesia/style.css” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:42 +0000] “GET /vps-ip-dns/?cf_action=sync_comments&post_id=499 HTTP/1.1” 200 48 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
71.96.108.116 - - [18/Sep/2013:06:56:46 +0000] “-” 400 0 “-” “-”
71.96.108.116 - - [18/Sep/2013:06:56:46 +0000] “-” 400 0 “-” “-”
71.96.108.116 - - [18/Sep/2013:06:57:34 +0000] “GET /dataguru-beijing-meeting-20130616/ HTTP/1.1” 200 8423 “http://blog.fens.me/vps-ip-dns/” “Mozilla/5.0 (Windows NT 6.2; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0”
66.249.66.84 - - [18/Sep/2013:06:57:35 +0000] “GET /series-angular/?cf_action=sync_comments&post_id=1991 HTTP/1.1” 200 48 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
124.42.13.230 - - [18/Sep/2013:06:57:48 +0000] “GET /mongodb-replica-set/ HTTP/1.1” 200 50131 “http://www.baidu.com/s?tn=baiduhome_pg&ie=utf-8&bs=%E5%9C%A8linux%E5%90%AF%E5%8A%A8%E4%B8%8Bmongodb.conf&f=8&rsv_bp=1&wd=about+to+fork+child+process%2C+waiting+until+server+is+ready+for+connections.&rsv_n=2&rsv_sug3=1&rsv_sug1=1&rsv_sug4=187&inputT=906” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:48 +0000] “GET /wp-content/themes/silesia/style.css HTTP/1.1” 200 34090 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:50 +0000] “GET /wp-content/themes/silesia/js/load.js HTTP/1.1” 200 715 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:50 +0000] “GET /wp-content/themes/silesia/functions/js/shortcode.js HTTP/1.1” 200 333 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:50 +0000] “GET /wp-content/themes/silesia/functions/css/shortcodes.css HTTP/1.1” 200 2899 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:50 +0000] “GET /wp-includes/js/jquery/jquery-migrate.min.js?ver=1.2.1 HTTP/1.1” 200 7200 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:50 +0000] “GET /wp-includes/js/comment-reply.min.js?ver=3.6 HTTP/1.1” 200 786 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:51 +0000] “GET /wp-content/themes/silesia/js/jquery.cycle.all.min.js HTTP/1.1” 200 31539 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:51 +0000] “GET /wp-includes/js/jquery/jquery.js?ver=1.10.2 HTTP/1.1” 200 93128 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /wp-content/themes/silesia/images/slide-bg.png HTTP/1.1” 200 934 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /js/google.js HTTP/1.1” 200 475 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /js/baidu.js HTTP/1.1” 200 249 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /wp-content/uploads/2013/05/rs.png HTTP/1.1” 200 35768 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /wp-content/themes/silesia/images/ico-twitter.png HTTP/1.1” 200 2128 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /wp-content/themes/silesia/images/crubms-div.png HTTP/1.1” 200 1255 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /wp-content/themes/silesia/images/home-ico.png HTTP/1.1” 200 1103 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”
124.42.13.230 - - [18/Sep/2013:06:57:53 +0000] “GET /wp-content/themes/silesia/images/sprites/post-type.png HTTP/1.1” 200 2009 “http://blog.fens.me/mongodb-replica-set/” “Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS101170; InfoPath.2; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727)”

WebLogMapper

public class WebLogMapper extends Mapper<LongWritable, Text,Text, NullWritable> {

    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        //1.获取一行数据
        String line = value.toString();

        //2.切割
        String[] words = StringUtils.split(line, ' ');

        //3.etl
        if (words.length >= 11){
            context.write(value,NullWritable.get());
        }

    }
}

WebLogDriver

public class WebLogDriver {
    public static void main(String[] args) throws Exception {

// 输入输出路径需要根据自己电脑上实际的输入输出路径设置
        args = new String[] { "D:\\MapReduce_Data_Test\\etl\\input", "D:\\MapReduce_Data_Test\\etl\\output" };

        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf);

        job.setJarByClass(WebLogDriver.class);

        job.setMapperClass(WebLogMapper.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(NullWritable.class);

        // 不需要reduceTask 设置reduceTask个数为0
        job.setNumReduceTasks(0);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        boolean b = job.waitForCompletion(true);
        System.exit(b ? 0 : 1);
    }
}

MapReduce总结

MapReduce
    1.InputFormat
        1)默认是TextInputFormat<LongWritable,Text> K:偏移量 V:一行文本内容
        2)处理小文件使用CombineTextInputFormat 把多个小文件合并在一起（逻辑上合并到一起）统一切片,也就是使得多个小文件交给一个MapTask去执行
        3)自定义InputFormat需要实现 isSplitable() 和 createRecordReader()方法
    2.Mapper
        setup()方法: 预处理方法(可以从这里获取当前处理的文件名) map():用户自定义的逻辑  cleanup():关闭资源
    3.分区Partitioner
        1)默认分区HashPartitioner 默认分区个数=reduceTask的个数,分区的规则是hash分区(分区号=key的hashcode值%reduceTask个数)
            hash分区实现了负载平衡
        2)自定义Partitioner,继承Partitioner类并重写getPartition()方法,自定义分区规则(分区号必须从0开始每次+1)
    4.排序
        1)部分排序: 从shuffle阶段输出的每个文件内部有序。
        2)全排序: 只有一个reduceTask就是全排序。
        3)二次排序: 自定义排序，实现 WritableComparable接口 重写compareTo方法，在自定义排序过程中，如果compareTo中的判断条件为两个即为二次排序。
    5.Combiner
        前提：不影响最终的业务逻辑(求和 )
        提前预聚合,减少网络带宽和数据大小
    6.Reducer
        用户的业务逻辑
        setup() 初始化
        reduce() 用户的业务逻辑
        cleanup() 关闭资源
    7.OutputFormat
        1)默认TextOutputFormat 按行输出到文件
        2)自定义OutputFormat

让线程再跑一会
关注
  • 1
    点赞
  • 踩
  • 6
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
专栏目录
【MapReduce篇07】MapReduce之数据清洗ETL1
08-04
【MapReduce篇07】MapReduce之数据清洗ETL1
Mapreduce_Map 数据清洗（ETL）案例实操
weixin_61959079的博客
06-22 340
使用Mapper对数据集进行清洗，业务逻辑写在Map阶段，能满足正常ETL。以上是我通过网络学习，自己总结和练习的过程。一是为了防止自己忘记学过的知识，二是分享自己学习过程得到的结果，以此来发布博客。以上如有雷同，请联系本人！
Hadoop（MapReduce数据清理、压缩）笔记06
2201_75649224的博客
09-10 290
1）输入数据接口：InputFormat（1）默认使用的实现类是：TextInputFormat（2）TextInputFormat 的功能逻辑是：一次读一行文本，然后将该行的起始偏移量作为key，行内容作为 value 返回。（3）CombineTextInputFormat 可以把多个小文件合并成一个切片处理，提高处理效率。2）逻辑处理接口：Mapper用户根据业务需求实现其中三个方法：map() setup() cleanup ()3）Partitioner 分区。
大数据采集、清洗、处理：使用MapReduce进行离线数据分析完整案例
热门推荐
leeyu的博客
04-23 2万+
1 大数据处理的常用方法 大数据处理目前比较流行的是两种方法，一种是离线处理，一种是在线处理，基本处理架构如下： 在互联网应用中，不管是哪一种处理方式，其基本的数据来源都是日志数据，例如对于web应用来说，则可能是用户的访问日志、用户的点击日志等。 如果对于数据的分析结果在时间上有比较严格的要求，则可以采用在线处理的方式来对数据进行分析，如使用Spark、Storm等进行处理。比较贴切的...
mapreduce清洗实战
qq_43414681的博客
09-25 884
mapreduce清洗json案例 目标：去重去空，清洗不符合规则的数据以及数据格式转换（num-numk·num薪转换成numk·12薪，数据只取第一个num-保留小数点后两位，如20-30k·18薪就转换成了37.50） #编写map public class LiePingMapper extends Mapper<LongWritable, Text, Text, NullWritable> { private static Text text =new Text();
【大数据之Hadoop】十七、MapReduce之数据清洗ETL
阿宁的博客
04-21 1819
ETL是将业务系统的数据经过抽取、清洗转换之后加载到数据仓库的过程，目的是将分散、零乱、标准不统一的数据整合到一起，为决策提供分析依据。ETL的设计分三部分：数据抽取、数据的清洗转换、数据的加载。
大数据技术
12-19
本阶段详细介绍了大数据所涉及到的Linux、shell、Hadoop、zookeeper、HadoopHA、Hive、Flume、Kafka、Hbase、Sqoop、Oozie等技术的概念、安装配置、架构原理、数据类型定义、数据操作、存储集群等重点知识点。
14 - MapReduce之数据清洗(ETL)案例,倒排索引案例,ReduceTask 工作机制,Hadoop 数据压缩简介
花&败
08-10 2320
一：简介 在运行核心业务 Mapreduce 程序之前，往往要先对数据进行清洗，清理掉不符合用户要求的数据。 清理的过程往往只需要运行 mapper 程序，不需要运行 reduce 程序。 二：日志清洗案例之简单解析版 需求：去除日志中字段长度小于等于11的日志(每一行按照空格切割，切割后数组长度小于11的日志不要) 数据如下： 代码实现......
【MapReduce】实现简单的数据清洗
HR的博客
11-20 7797
MapReduce--->实现简单的数据清洗题目和要求题目第一小题Map阶段Reduce阶段Driver阶段第二小题Map阶段Reduce阶段Driver阶段 题目和要求 题目 2020年新冠肺炎对我国社会各方面影响巨大，大数据技术在抗击疫情过程中发挥了巨大作用，尤其在新增、确认等相关病例数据的采集及统计上应用颇广，下面有一份数据是今年1月20-4月29日的全国各省市及国外的疫情数据，请你按照要求使用MapReduce程序完成相关数据预处理。 第一小题 Map阶段 Reduce阶段 Driver阶
第1关：数据清洗MapReduce综合应用案例 — 招聘数据清洗
qq_61604164的博客
11-15 7948
第1关：数据清洗 MapReduce综合应用案例 — 招聘数据清洗
mapreduce项目 数据清洗
10-10
mapreduce基本数据读取，通俗易懂。 此项目情景为，遗传关系族谱。 （爷爷 、父母 、孩子） 经典案例
MapReduce 实践数据
01-28
MapReduce章节wordcount 实践材料，大数据学习实践材料。
MapReduce综合应用案例 - 招聘数据清洗-结果
04-02
MapReduce综合应用案例 - 招聘数据清洗-结果
E-MapReduce极客挑战赛季军方案
gangyikeji的博客
04-22 590
前一段时间我参加了E-MapReduce极客挑战赛，很幸运的获得了季军。在这把我的比赛攻略给大家分享一下，希望可以抛砖引玉。
【ROMA解决方案和功能架构】
数据也是生产力，保持热爱，奔赴山海！
04-23 800
1、基于ROMA融合集成平台，实现企业应用、数据、API 、OT设备、云服务、合作伙伴应用之间的集成连接。2、ROMA主要包含四个组件：数据集成（Fast Data Integration，简称FDI）、服务集成（API Connect，简称APIC）、消息集成（Message Queue Service，简称MQS）、设备集成（LINK）。
生产数据采集系统
weixin_50072277的博客
04-23 695
在数字化浪潮的推动下，生产数据采集系统已经成为企业提升生产效率、优化运营管理的关键工具。那么，什么是生产数据采集系统呢？简单来说，生产数据采集系统是指通过一系列技术手段，实时收集、处理和分析生产线上的各类数据，从而为企业的决策提供有力支持的系统。
哪些情况下左联接的数据量比左表大？
Java/Python/大数据干货分享
04-26 348
本文主要探讨SQL和Pandas左连接数据量比左表大的问题。在某些情况下，执行左连接时，SQL和Pandas输出的结果会比左表数据量大。这是为什么呢？左连接是关系型数据库中最常见的连接之一。它的作用是从左表中取出所有的行，以及与右表匹配的所有行。如果某个行在右表中没有匹配到，则会在输出结果中以NULL的形式呈现1）左表与右表为一对多或多对多2）左表与右表关联键都存在空值NULL1）视情景删除或聚合右表中重复关联键的数据2）过滤表中关联键值为NULL的行。
25计算机考研院校数据分析 | 电子科技大学
最新发布
m0_72717598的博客
04-29 412
是国家“985工程”、“211工程”、“世界一流大学和一流学科”重点建设高校，入选“2011计划”、“111计划”、“卓越工程师教育培养计划”、“国家建设高水平大学公派研究生项目”、“中国政府奖学金来华留学生接收院校”，两电一邮成员。题库中抽取题目作答，重点考查程序设计、算法设计与分析、软件工程、计算机组成原理、数据库、计算机网络、编译原理、密码学（该科目限报考083900网络空间安全专业考生）等基础知识情况。101思想政治理论、201英语一、301数学一、820计算机专业基础【22数据结构+操作系统】
mapreduce数据清洗
04-07
MapReduce数据清洗是指利用MapReduce框架进行大规模数据处理时，对数据进行预处理和清洗，以便更好地进行后续分析和应用。 数据清洗是数据处理流程中的重要环节，因为原始数据可能存在格式不一致、缺失值、错误值等...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

让线程再跑一会

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付：¥1
获取中
扫码支付

您的余额不足，请更换扫码支付或充值

打赏作者

实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值