目录
今天我们将Hadoop技术应用到了一个电商数据分析项目中。通过分析用户行为数据，我们尝试找出潜在的商业价值。这个项目不仅锻炼了我的技术能力，也提高了我的团队协作能力。
1.项目要求
根据电商日志文件，分析：
-
统计页面浏览量（每行记录就是一次浏览）
-
统计各个省份的浏览量 （需要解析IP）
-
日志的ETL操作（ETL：数据从来源端经过抽取（Extract）、转换（Transform）、加载（Load）至目的端的过程）
最后思考为什么要ETL：没有必要解析出所有数据，只需要解析出有价值的字段即可。本项目中需要解析出：ip、url、pageId（topicId对应的页面Id）、country、province、city。
2.项目数据（部分）
20946835322http://www.yihaodian.com/1/?tracker_u=2225501&type=3http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=42351号店1SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,unionKey:22255012013-07-21 09:30:01\Nhttp://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=42351\Nnull-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32上海市12013-07-21 09:30:01上海市66\N\N\N\N2013-07-21
20947290187http://www.yihaodian.com/ctg/s2/c5287-%E5%A4%A7%E7%B1%B3/b/a28186,28184-s1-v0-p1-price-d0-f0-m1-rt0-pid-k/1/http://www.yihaodian.com/ctg/s2/c5287-%E5%A4%A7%E7%B1%B3/b1879/a281862SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU22255011124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:32:30134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=4235110\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32 93 1search_order_2上海市1null上海市66\N\N\N\N2013-07-21
20947323490http://www.yihaodian.com/cmsPage/show.do?pageId=49248&provinceId=1http://www.yihaodian.com/cmsPage/show.do?pageId=48963&provinceId=13SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:37:04134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423520\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32cms_pic_48963_542760_0上海市1上海市66\N\N\N\N2013-07-21
20947273602https://passport.yihaodian.com/passport/login_input.dohttp://www.yihaodian.com/1/?tracker_u=2225501&type=33SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,unionKey:22255012013-07-21 09:30:11\Nhttp://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=42353\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32YHD_TOP_login上海市1上海市66\N\N\N\N2013-07-21
20947308318http://www.yihaodian.com/marketing/allproduct.htmlhttp://www.yihaodian.com/item/user/continueShopping.do?ref=detail_continue_shopping_B-1_1_52_search.ctg_13SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:34:59134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423516\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32YHD_TOP_index_5上海市1上海市66\N\N\N\N2013-07-21
20948276566http://www.yihaodian.com/checkoutV3/index.dohttp://www.yihaodian.com/cart/cart.do?action=view3SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:50:43134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423567\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32mobile上海市1上海市66\N\N\N\N2013-07-21
20948301864http://www.yihaodian.com/checkoutV3/index.dohttp://www.yihaodian.com/cart/cart.do?action=view3SKAPHD3JZYH9EE9ACB1NGA9VDQHNJMX1NY9TPPG4SWG71358HGRJGQHQQBXY9GF96CVU2225501\N124.79.172.232msessionid:YR9H5YU7RZ8Y94EBJNZ2P5W8DT37Q9JH,uname:chen45311,unionKey:22255012013-07-21 09:53:55134885852http://www.baidu.com/s?wd=1%E5%8F%B7%E5%BA%97&rsv_bp=0&ch=&tn=baidu&bar=&rsv_spt=3&ie=utf-8&rsv_sug3=5&rsv_sug=0&rsv_sug1=4&rsv_sug4=313&inputT=423570\N1-10Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MATP; Media Center PC 6.0; .NET4.0C; InfoPath.2; .NET4.0E)Win32packagecontent上海市1上海市66\N\N\N\N2013-07-21
20979872853http://www.yihaodian.com/1/?type=3&tracker_u=109740492583BAWG49VCYYTMZ6VU9XX74KPV5CCHPAQ2A4A5PPG68XWJNUSSX649S4YQTCT6HBMS9KBA10974049258\N27.45.216.128,unionKey:109740492582013-07-21 18:58:21\N1\Nnull247Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)Win32广东省20惠州市\N\N\N\N2013-07-21
20977690754http://www.yihaodian.com/1/?type=3&tracker_u=10781349260349FDCP696X2RDDRC2ED6Y4JVPTEVFNDADF1DPPGCTKD92UT3DR7KY1VFZ92ZU4HEP47910781349260\N101.80.223.6,unionKey:107813492602013-07-21 18:11:46\N1\Nnull-10Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)Win32上海市12013-07-21 18:11:46上海市\N\N\N\N2013-07-21
20971340675http://m.yihaodian.com/mw/getcity/2?fromid=2&provinceId_temp2=&goodReceiverId=-1&provinceId_temp=2&cityId_temp=53&countyid=53&address1=&receiveName=&receiverMobile=&receiverPhone=395AKDN3KREQA951229CZ9KR47N7CSCSBF6EKPPGCUGWV5P58N7RE1SJQVB1488974NTD\N111.30.132.1942013-07-21 16:29:55\N1\Nnull-205.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.13 (KHTMLLinux i686北京市22013-07-21 16:29:55北京市\N\N\N\N2013-07-21
20939109069http://www.yihaodian.com/1/?type=3&tracker_u=107813492603YU7317CWPWJPCCKSJD83RZ35W6F1DGHT4P93PPGDPBNYMTMMVQQSX13BVW4A6Z61BCFK10781349260\N115.28.44.205msessionid:HTEVTDB9S35AXCX54SWZ28XBJ11WWSA3,unionKey:107813492602013-07-21 01:19:29\N2\Nnull-20Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Sicent; WoShiHoney.B; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; Microsoft Windows Media Center PC 6.0)Win32北京市2北京市\N\N\N\N2013-07-21
20939403234http://www.yihaodian.com/1/?type=3&tracker_u=102719223923YU7317CWPWJPCCKSJD83RZ35W6F1DGHT4P93PPGDPBNYMTMMVQQSX13BVW4A6Z61BCFK10271922392\N115.28.44.205,unionKey:102719223922013-07-21 01:26:07\N3\Nnull-20Mozilla/4.0 (compatible; MSIE 8.0; YYGameAll_1.2.167057.92; Windows NT 5.1; Trident/4.0; QQDownload 718; AskTbSPC2/5.13.1.18107; .NET4.0C; .NET4.0E)Win32北京市2北京市\N\N\N\N2013-07-21
20939177966http://www.yihaodian.com/1/?type=3&tracker_u=107813492603YU7317CWPWJPCCKSJD83RZ35W6F1DGHT4P93PPGDPBNYMTMMVQQSX13BVW4A6Z61BCFK10781349260\N115.28.44.205,unionKey:107813492602013-07-21 01:18:59\N1\Nnull-20Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Sicent; WoShiHoney.B; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; Microsoft Windows Media Center PC 6.0)Win32北京市2北京市\N\N\N\N2013-07-21
20966958616http://www.1mall.com/item/11033978_4http://www.1mall.com/item/11033980_4?ref=1_1_51_search.keyword_13EUBCC9CG8G7G3Z331PWEV3UNWD7QZA1F3Q2Q1[{1=[{9784616=38251:1190}]}]PPH24C5RHH3GE4RUY1UKQW84PVYBXZFT7439964\N101.18.106.107msessionid:CP23DVRKZGQ67MWQQGNHN5USRK2R9W8V,unionKey:74399642013-07-21 15:11:16\Nhttp://www.2345.com/?kskypw59784615\N4-1Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTMLWin32河北省446\N\N\N\N2013-07-21
20966773971http://search.yihaodian.com/s2/c0-0/k%25E9%259B%25AA%25E7%25BA%25BA%2520t%25E6%2581%25A4/1/http://www.yihaodian.com/4/?tracker_u=7439964&type=13EUBCC9CG8G7G3Z331PWEV3UNWD7QZA1F3Q2QPPH24C5RHH3GE4RUY1UKQW84PVYBXZFT7439964\N101.18.106.107msessionid:WUEYCTYUHP7CUP4UD7Y2VHNZABPCYJ18,unionKey:74399642013-07-21 15:10:02\Nhttp://www.2345.com/?kskypw3\N4-1Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTMLWin32雪纺 t恤13661CatMenu_Keywords_100000003_18258河北省4null3\N\N\N\N2013-07-21
20966425286http://www.yihaodian.com/4/?tracker_u=7439964&type=1http://www.2345.com/?kskypw3EUBCC9CG8G7G3Z331PWEV3UNWD7QZA1F3Q2QPPH24C5RHH3GE4RUY1UKQW84PVYBXZFT7439964\N101.18.106.107msessionid:WUEYCTYUHP7CUP4UD7Y2VHNZABPCYJ18,unionKey:74399642013-07-21 15:09:39\Nhttp://www.2345.com/?kskypw1\N4-1Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTMLWin32河北省43\N\N\N\N2013-07-21
20966253161http://www.yihaodian.com/4/?tracker_u=7439964&type=1http://www.2345.com/?kskypw3EUBCC9CG8G7G3Z331PWEV3UNWD7QZA1F3Q2QPPH24C5RHH3GE4RUY1UKQW84PVYBXZFT7439964\N101.18.106.107msessionid:WUEYCTYUHP7CUP4UD7Y2VHNZABPCYJ18,unionKey:74399642013-07-21 15:09:41\Nhttp://www.2345.com/?kskypw2\N4-1Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTMLWin32YHD_HOME_HTAB_BRANDSHOP河北省43\N\N\N\N2013-07-21
20966870026http://www.1mall.com/item/11033980_4?ref=1_1_51_search.keyword_1http://search.yihaodian.com/s2/c0-0/k%25E9%259B%25AA%25E7%25BA%25BA%2520t%25E6%2581%25A4/1/3EUBCC9CG8G7G3Z331PWEV3UNWD7QZA1F3Q2Q1[{1=[{9784617=38251:1200}]}]PPH24C5RHH3GE4RUY1UKQW84PVYBXZFT7439964\N101.18.106.107msessionid:CP23DVRKZGQ67MWQQGNHN5USRK2R9W8V2013-07-21 15:11:05\Nhttp://www.2345.com/?kskypw49784615\N4-1Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.1 (KHTMLWin32pro_31河北省4\N\N\N\N2013-07-21
20983556019http://www.yihaodian.com/1/?tracker_u=10861423206&type=1http://www.hao123.com/?tn=44059078_1_hao_pg339PJC7MXTDPP1313EJFZY89FMV7CEN48Q37YPPH8E24JEEAQMC2V2XYGJ4GV5P1862DG10861423206\N61.227.170.83msessionid:N8PXM3Q34ZW4NY9S2CYEF1D2QH14U29Y,unionKey:108614232062013-07-21 20:03:30\Nhttp://www.hao123.com/?tn=44059078_1_hao_pg1\Nnull-1Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0Win32台湾312013-07-21 20:03:3018\N\N\N\N2013-07-21
20991369746http://t.1mall.com/2?grouponAreaId=37http://t.1mall.com/100?grouponAreaId=373Y8GC5H8GDMWV9DRHGEZ9P54G1QSS8VUZEXX5PPHEYNNKGRP5A1XDWBU624CA1TWAUYPE10861423206\N219.137.131.9msessionid:NENZXUYMUKAGT2URMPJXS3969RQNGHJC,uname:liwdo,unionKey:108614232062013-07-21 22:00:36115138574http://t.1mall.com/3?grouponAreaId=37&sortType=1&page=24\N20237Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTMLWin32YHD_GROUPON_MAIN_NAV_2广东省20广州市67\N\N\N\N2013-07-21
20991255201http://mingpin.1mall.com/3/http://t.1mall.com/100?grouponAreaId=373Y8GC5H8GDMWV9DRHGEZ9P54G1QSS8VUZEXX5PPHEYNNKGRP5A1XDWBU624CA1TWAUYPE10861423206\N219.137.131.9msessionid:NENZXUYMUKAGT2URMPJXS3969RQNGHJC,uname:liwdo,unionKey:108614232062013-07-21 21:59:49115138574http://t.1mall.com/3?grouponAreaId=37&sortType=1&page=23\N20237Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTMLWin325823_647270_5广东省20广州市67\N\N\N\N2013-07-21
20990627919http://t.1mall.com/100?grouponAreaId=37http://mingpin.1mall.com/3/3Y8GC5H8GDMWV9DRHGEZ9P54G1QSS8VUZEXX5PPHEYNNKGRP5A1XDWBU624CA1TWAUYPE10861423206\N219.137.131.9msessionid:NENZXUYMUKAGT2URMPJXS3969RQNGHJC,uname:liwdo,unionKey:108614232062013-07-21 21:59:42115138574http://t.1mall.com/3?grouponAreaId=37&sortType=1&page=22\N20237Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTMLWin325823_651107_3广东省20广州市67\N\N\N\N2013-07-21
20991148095http://mingpin.1mall.com/3/http://t.1mall.com/3?grouponAreaId=37&sortType=1&page=23Y8GC5H8GDMWV9DRHGEZ9P54G1QSS8VUZEXX5PPHEYNNKGRP5A1XDWBU624CA1TWAUYPE10861423206\N219.137.131.9msessionid:NENZXUYMUKAGT2URMPJXS3969RQNGHJC,uname:liwdo,unionKey:108614232062013-07-21 21:59:34115138574http://t.1mall.com/3?grouponAreaId=37&sortType=1&page=21\N20237Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTMLWin32YHD_GROUPON_MAIN_NAV_MP广东省20广州市67\N\N\N\N2013-07-21
20957021694http://www.yihaodian.com/1/?type=1http://www.airenti.org/Html/Disp/4551_1.html3WE8VM6AEQFDG5Q6FVWJVYP1G564UTG7VF3SSPPHG5ESE7XNCG76BBTYDTE1M3RJ15N9N\N125.39.238.86msessionid:9158PM2GVNYKKF1G7MJSQPU2ZRB6T5C42013-07-21 12:24:59\Nhttp://www.airenti.org/Html/Disp/4551_1.html1\Nnull-30Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTMLWin32天津市3天津市15\N\N\N\N2013-07-21
20960979033http://www.yihaodian.com/1/?tracker_u=10931041909&type=1http://hao.360.cn/?wd_xp136ZD1N3J3ECTNX96DSGESX9GN12U1JD2R9YGPPPHK3755F3XK3HDYT1AHW7XNS9GZBECK10931041909\N101.85.27.156msessionid:D7UNQY44Z6GYKXHWB7QW8NMT6TPVQZKK,unionKey:109310419092013-07-21 13:32:38\Nhttp://hao.360.cn/?wd_xp11\Nnull-10Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTMLWin32上海市12013-07-21 13:32:38上海市4\N\N\N\N2013-07-21
20960991758http://www.yihaodian.com/cms/view.do?topicId=19004http://www.yihaodian.com/cms/view.do?topicId=22331&cache=false&merchant=136ZD1N3J3ECTNX96DSGESX9GN12U1JD2R9YGPPPHK3755F3XK3HDYT1AHW7XNS9GZBECK10931041909\N101.85.27.156msessionid:D7UNQY44Z6GYKXHWB7QW8NMT6TPVQZKK,unionKey:109310419092013-07-21 13:34:10\Nhttp://hao.360.cn/?wd_xp15\N1-10Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)Win32上海市1上海市4\N\N\N\N2013-07-21
20960982182http://www.yihaodian.com/cms/view.do?topicId=1899436ZD1N3J3ECTNX96DSGESX9GN12U1JD2R9YGPPPHK3755F3XK3HDYT1AHW7XNS9GZBECK10931041909\N101.85.27.156msessionid:D7UNQY44Z6GYKXHWB7QW8NMT6TPVQZKK,unionKey:109310419092013-07-21 13:33:00\Nhttp://hao.360.cn/?wd_xp13\N1-10Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)Win329075_1092661_5上海市1上海市4\N\N\N\N2013-07-21
20960815581http://www.yihaodian.com/1/?tracker_u=10931041909&type=1http://www.yihaodian.com/1/?tracker_u=10931041909&type=136ZD1N3J3ECTNX96DSGESX9GN12U1JD2R9YGPPPHK3755F3XK3HDYT1AHW7XNS9GZBECK10931041909\N101.85.27.156msessionid:D7UNQY44Z6GYKXHWB7QW8NMT6TPVQZKK,unionKey:109310419092013-07-21 13:32:40\Nhttp://hao.360.cn/?wd_xp12\N1-10Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTMLWin32上海市1上海市4\N\N\N\N2013-07-213.项目代码
3.1.ETLApp.java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
import java.util.Map;
public class ETLApp {
public static void main(String[] args) throws Exception{
Configuration configuration = new Configuration();
FileSystem fileSystem = FileSystem.get(configuration);
Path outputPath = new Path(args[1]);
if (fileSystem.exists(outputPath)) {
fileSystem.delete(outputPath, true);
}
Job job = Job.getInstance(configuration);
job.setJarByClass(ETLApp.class);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(NullWritable.class);
job.setMapOutputValueClass(Text.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
static class MyMapper extends Mapper<LongWritable, Text, NullWritable, Text> {
private LongWritable ONE = new LongWritable(1);
private LogParser logParser;
@Override
protected void setup(Context context) throws IOException, InterruptedException {
logParser = new LogParser();
}
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String log = value.toString();
Map<String, String> info = logParser.parseV2(log);
String ip=info.get("ip");
String country=info.get("country");
String province=info.get("province");
String city=info.get("city");
String url=info.get("url");
String time=info.get("time");
String pageId=ContentUtils.getPageId(url);
StringBuilder builder=new StringBuilder();
builder.append(ip).append("\t");
builder.append(country).append("\t");
builder.append(province).append("\t");
builder.append(city).append("\t");
builder.append(url).append("\t");
builder.append(time).append("\t");
builder.append(pageId);
context.write(NullWritable.get(),new Text(builder.toString()));
}
}
}
3.2.GetPageld.java
import org.apache.commons.lang.StringUtils;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class GetPageId {
public static String getPageId(String url) {
String pageId = "";
if (StringUtils.isBlank(url)) {
return pageId;
}
Pattern pat = Pattern.compile("topicId=[0-9]+");
Matcher matcher = pat.matcher(url);
if (matcher.find()) {
pageId = matcher.group().split("topicId=")[1];
}
return pageId;
}
public static void main(String[] args) {
System.out.println(getPageId("http://www.yihaodian.com/cms/view.do?topicId=14572"));
System.out.println(getPageId("http://www.yihaodian.com/cms/view.do?topicId=22372&merchant=1"));
}
}
3.3.IPParser.java
public class IPParser extends IPSeeker {
// 地址 仅仅只是在ecplise环境中使用，部署在服务器上，需要先将qqwry.dat放在集群的各个节点某个有读取权限目录，
// 然后在这里指定全路径
private static final String ipFilePath = "ip/qqwry.dat";
// 部署在服务器上
//private static final String ipFilePath = "/opt/datas/qqwry.dat";
private static IPParser obj = new IPParser(ipFilePath);
protected IPParser(String ipFilePath) {
super(ipFilePath);
}
public static IPParser getInstance() {
return obj;
}
public RegionInfo analyseIp(String ip) {
if (ip == null || "".equals(ip.trim())) {
return null;
}
RegionInfo info = new RegionInfo();
try {
String country = super.getCountry(ip);
if ("局域网".equals(country) || country == null || country.isEmpty() || country.trim().startsWith("CZ88")) {
// 设置默认值
info.setCountry("中国");
info.setProvince("上海市");
} else {
int length = country.length();
int index = country.indexOf('省');
if (index > 0) { // 表示是国内的某个省
info.setCountry("中国");
info.setProvince(country.substring(0, Math.min(index + 1, length)));
int index2 = country.indexOf('市', index);
if (index2 > 0) {
// 设置市
info.setCity(country.substring(index + 1, Math.min(index2 + 1, length)));
}
} else {
String flag = country.substring(0, 2);
switch (flag) {
case "内蒙":
info.setCountry("中国");
info.setProvince("内蒙古自治区");
country = country.substring(3);
if (country != null && !country.isEmpty()) {
index = country.indexOf('市');
if (index > 0) {
// 设置市
info.setCity(country.substring(0, Math.min(index + 1, length)));
}
// TODO:针对其他旗或者盟没有进行处理
}
break;
case "广西":
case "西藏":
case "宁夏":
case "新疆":
info.setCountry("中国");
info.setProvince(flag);
country = country.substring(2);
if (country != null && !country.isEmpty()) {
index = country.indexOf('市');
if (index > 0) {
// 设置市
info.setCity(country.substring(0, Math.min(index + 1, length)));
}
}
break;
case "上海":
case "北京":
case "重庆":
case "天津":
info.setCountry("中国");
info.setProvince(flag + "市");
country = country.substring(3);
if (country != null && !country.isEmpty()) {
index = country.indexOf('区');
if (index > 0) {
// 设置市
char ch = country.charAt(index - 1);
if (ch != '小' || ch != '校') {
info.setCity(country.substring(0, Math.min(index + 1, length)));
}
}
if ("unknown".equals(info.getCity())) {
// 现在city还没有设置，考虑县
index = country.indexOf('县');
if (index > 0) {
// 设置市
info.setCity(country.substring(0, Math.min(index + 1, length)));
}
}
}
break;
case "香港":
case "澳门":
info.setCountry("中国");
info.setProvince(flag + "特别行政区");
break;
default:
info.setCountry(country); // 针对其他国外的ip
}
}
}
} catch (Exception e) {
// nothing
}
return info;
}
/**
* ip地址对应的info类
*
*/
public static class RegionInfo {
private String country ;
private String province ;
private String city ;
public String getCountry() {
return country;
}
public void setCountry(String country) {
this.country = country;
}
public String getProvince() {
return province;
}
public void setProvince(String province) {
this.province = province;
}
public String getCity() {
return city;
}
public void setCity(String city) {
this.city = city;
}
@Override
public String toString() {
return "RegionInfo [country=" + country + ", province=" + province + ", city=" + city + "]";
}
}
}
3.4.IPSeeker.java
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.io.UnsupportedEncodingException;
import java.nio.ByteOrder;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.util.ArrayList;
import java.util.Hashtable;
import java.util.List;
public class IPSeeker {
public static final String ERROR_RESULT = "错误的IP数据库文件";
// 一些固定常量，比如记录长度等等
private static final int IP_RECORD_LENGTH = 7;
private static final byte AREA_FOLLOWED = 0x01;
private static final byte NO_AREA = 0x2;
// 用来做为cache，查询一个ip时首先查看cache，以减少不必要的重复查找
private Hashtable ipCache;
// 随机文件访问类
private RandomAccessFile ipFile;
// 内存映射文件
private MappedByteBuffer mbb;
// 单一模式实例
private static IPSeeker instance = null;
// 起始地区的开始和结束的绝对偏移
private long ipBegin, ipEnd;
// 为提高效率而采用的临时变量
private IPLocation loc;
private byte[] buf;
private byte[] b4;
private byte[] b3;
/** */
/**
* 私有构造函数
*/
protected IPSeeker(String ipFilePath) {
ipCache = new Hashtable();
loc = new IPLocation();
buf = new byte[100];
b4 = new byte[4];
b3 = new byte[3];
try {
ipFile = new RandomAccessFile(ipFilePath, "r");
} catch (FileNotFoundException e) {
System.out.println("IP地址信息文件没有找到，IP显示功能将无法使用");
ipFile = null;
}
// 如果打开文件成功，读取文件头信息
if (ipFile != null) {
try {
ipBegin = readLong4(0);
ipEnd = readLong4(4);
if (ipBegin == -1 || ipEnd == -1) {
ipFile.close();
ipFile = null;
}
} catch (IOException e) {
System.out.println("IP地址信息文件格式有错误，IP显示功能将无法使用");
ipFile = null;
}
}
}
/** */
/**
* @return 单一实例
*/
public static IPSeeker getInstance(String ipFilePath) {
if (instance == null) {
instance = new IPSeeker(ipFilePath);
}
return instance;
}
/** */
/**
* 给定一个地点的不完全名字，得到一系列包含s子串的IP范围记录
*
* @param s
* 地点子串
* @return 包含IPEntry类型的List
*/
public List getIPEntriesDebug(String s) {
List ret = new ArrayList();
long endOffset = ipEnd + 4;
for (long offset = ipBegin + 4; offset <= endOffset; offset += IP_RECORD_LENGTH) {
// 读取结束IP偏移
long temp = readLong3(offset);
// 如果temp不等于-1，读取IP的地点信息
if (temp != -1) {
IPLocation loc = getIPLocation(temp);
// 判断是否这个地点里面包含了s子串，如果包含了，添加这个记录到List中，如果没有，继续
if (loc.country.indexOf(s) != -1 || loc.area.indexOf(s) != -1) {
IPEntry entry = new IPEntry();
entry.country = loc.country;
entry.area = loc.area;
// 得到起始IP
readIP(offset - 4, b4);
entry.beginIp = IPSeekerUtils.getIpStringFromBytes(b4);
// 得到结束IP
readIP(temp, b4);
entry.endIp = IPSeekerUtils.getIpStringFromBytes(b4);
// 添加该记录
ret.add(entry);
}
}
}
return ret;
}
/** */
/**
* 给定一个地点的不完全名字，得到一系列包含s子串的IP范围记录
*
* @param s
* 地点子串
* @return 包含IPEntry类型的List
*/
public List getIPEntries(String s) {
List ret = new ArrayList();
try {
// 映射IP信息文件到内存中
if (mbb == null) {
FileChannel fc = ipFile.getChannel();
mbb = fc.map(FileChannel.MapMode.READ_ONLY, 0, ipFile.length());
mbb.order(ByteOrder.LITTLE_ENDIAN);
}
int endOffset = (int) ipEnd;
for (int offset = (int) ipBegin + 4; offset <= endOffset; offset += IP_RECORD_LENGTH) {
int temp = readInt3(offset);
if (temp != -1) {
IPLocation loc = getIPLocation(temp);
// 判断是否这个地点里面包含了s子串，如果包含了，添加这个记录到List中，如果没有，继续
if (loc.country.indexOf(s) != -1 || loc.area.indexOf(s) != -1) {
IPEntry entry = new IPEntry();
entry.country = loc.country;
entry.area = loc.area;
// 得到起始IP
readIP(offset - 4, b4);
entry.beginIp = IPSeekerUtils.getIpStringFromBytes(b4);
// 得到结束IP
readIP(temp, b4);
entry.endIp = IPSeekerUtils.getIpStringFromBytes(b4);
// 添加该记录
ret.add(entry);
}
}
}
} catch (IOException e) {
System.out.println(e.getMessage());
}
return ret;
}
/** */
/**
* 从内存映射文件的offset位置开始的3个字节读取一个int
*
* @param offset
* @return
*/
private int readInt3(int offset) {
mbb.position(offset);
return mbb.getInt() & 0x00FFFFFF;
}
/** */
/**
* 从内存映射文件的当前位置开始的3个字节读取一个int
*
* @return
*/
private int readInt3() {
return mbb.getInt() & 0x00FFFFFF;
}
/** */
/**
* 根据IP得到国家名
*
* @param ip
* ip的字节数组形式
* @return 国家名字符串
*/
public String getCountry(byte[] ip) {
// 检查ip地址文件是否正常
if (ipFile == null)
return ERROR_RESULT;
// 保存ip，转换ip字节数组为字符串形式
String ipStr = IPSeekerUtils.getIpStringFromBytes(ip);
// 先检查cache中是否已经包含有这个ip的结果，没有再搜索文件
if (ipCache.containsKey(ipStr)) {
IPLocation loc = (IPLocation) ipCache.get(ipStr);
return loc.country;
} else {
IPLocation loc = getIPLocation(ip);
ipCache.put(ipStr, loc.getCopy());
return loc.country;
}
}
/** */
/**
* 根据IP得到国家名
*
* @param ip
* IP的字符串形式
* @return 国家名字符串
*/
public String getCountry(String ip) {
return getCountry(IPSeekerUtils.getIpByteArrayFromString(ip));
}
/** */
/**
* 根据IP得到地区名
*
* @param ip
* ip的字节数组形式
* @return 地区名字符串
*/
public String getArea(byte[] ip) {
// 检查ip地址文件是否正常
if (ipFile == null)
return ERROR_RESULT;
// 保存ip，转换ip字节数组为字符串形式
String ipStr = IPSeekerUtils.getIpStringFromBytes(ip);
// 先检查cache中是否已经包含有这个ip的结果，没有再搜索文件
if (ipCache.containsKey(ipStr)) {
IPLocation loc = (IPLocation) ipCache.get(ipStr);
return loc.area;
} else {
IPLocation loc = getIPLocation(ip);
ipCache.put(ipStr, loc.getCopy());
return loc.area;
}
}
/**
* 根据IP得到地区名
*
* @param ip
* IP的字符串形式
* @return 地区名字符串
*/
public String getArea(String ip) {
return getArea(IPSeekerUtils.getIpByteArrayFromString(ip));
}
/** */
/**
* 根据ip搜索ip信息文件，得到IPLocation结构，所搜索的ip参数从类成员ip中得到
*
* @param ip
* 要查询的IP
* @return IPLocation结构
*/
public IPLocation getIPLocation(byte[] ip) {
IPLocation info = null;
long offset = locateIP(ip);
if (offset != -1)
info = getIPLocation(offset);
if (info == null) {
info = new IPLocation();
info.country = "未知国家";
info.area = "未知地区";
}
return info;
}
/**
* 从offset位置读取4个字节为一个long，因为java为big-endian格式，所以没办法 用了这么一个函数来做转换
*
* @param offset
* @return 读取的long值，返回-1表示读取文件失败
*/
private long readLong4(long offset) {
long ret = 0;
try {
ipFile.seek(offset);
ret |= (ipFile.readByte() & 0xFF);
ret |= ((ipFile.readByte() << 8) & 0xFF00);
ret |= ((ipFile.readByte() << 16) & 0xFF0000);
ret |= ((ipFile.readByte() << 24) & 0xFF000000);
return ret;
} catch (IOException e) {
return -1;
}
}
/**
* 从offset位置读取3个字节为一个long，因为java为big-endian格式，所以没办法 用了这么一个函数来做转换
*
* @param offset
* @return 读取的long值，返回-1表示读取文件失败
*/
private long readLong3(long offset) {
long ret = 0;
try {
ipFile.seek(offset);
ipFile.readFully(b3);
ret |= (b3[0] & 0xFF);
ret |= ((b3[1] << 8) & 0xFF00);
ret |= ((b3[2] << 16) & 0xFF0000);
return ret;
} catch (IOException e) {
return -1;
}
}
/**
* 从当前位置读取3个字节转换成long
*
* @return
*/
private long readLong3() {
long ret = 0;
try {
ipFile.readFully(b3);
ret |= (b3[0] & 0xFF);
ret |= ((b3[1] << 8) & 0xFF00);
ret |= ((b3[2] << 16) & 0xFF0000);
return ret;
} catch (IOException e) {
return -1;
}
}
/**
* 从offset位置读取四个字节的ip地址放入ip数组中，读取后的ip为big-endian格式，但是
* 文件中是little-endian形式，将会进行转换
*
* @param offset
* @param ip
*/
private void readIP(long offset, byte[] ip) {
try {
ipFile.seek(offset);
ipFile.readFully(ip);
byte temp = ip[0];
ip[0] = ip[3];
ip[3] = temp;
temp = ip[1];
ip[1] = ip[2];
ip[2] = temp;
} catch (IOException e) {
System.out.println(e.getMessage());
}
}
/**
* 从offset位置读取四个字节的ip地址放入ip数组中，读取后的ip为big-endian格式，但是
* 文件中是little-endian形式，将会进行转换
*
* @param offset
* @param ip
*/
private void readIP(int offset, byte[] ip) {
mbb.position(offset);
mbb.get(ip);
byte temp = ip[0];
ip[0] = ip[3];
ip[3] = temp;
temp = ip[1];
ip[1] = ip[2];
ip[2] = temp;
}
/**
* 把类成员ip和beginIp比较，注意这个beginIp是big-endian的
*
* @param ip
* 要查询的IP
* @param beginIp
* 和被查询IP相比较的IP
* @return 相等返回0，ip大于beginIp则返回1，小于返回-1。
*/
private int compareIP(byte[] ip, byte[] beginIp) {
for (int i = 0; i < 4; i++) {
int r = compareByte(ip[i], beginIp[i]);
if (r != 0)
return r;
}
return 0;
}
/**
* 把两个byte当作无符号数进行比较
*
* @param b1
* @param b2
* @return 若b1大于b2则返回1，相等返回0，小于返回-1
*/
private int compareByte(byte b1, byte b2) {
if ((b1 & 0xFF) > (b2 & 0xFF)) // 比较是否大于
return 1;
else if ((b1 ^ b2) == 0)// 判断是否相等
return 0;
else
return -1;
}
/**
* 这个方法将根据ip的内容，定位到包含这个ip国家地区的记录处，返回一个绝对偏移 方法使用二分法查找。
*
* @param ip
* 要查询的IP
* @return 如果找到了，返回结束IP的偏移，如果没有找到，返回-1
*/
private long locateIP(byte[] ip) {
long m = 0;
int r;
// 比较第一个ip项
readIP(ipBegin, b4);
r = compareIP(ip, b4);
if (r == 0)
return ipBegin;
else if (r < 0)
return -1;
// 开始二分搜索
for (long i = ipBegin, j = ipEnd; i < j;) {
m = getMiddleOffset(i, j);
readIP(m, b4);
r = compareIP(ip, b4);
// log.debug(Utils.getIpStringFromBytes(b));
if (r > 0)
i = m;
else if (r < 0) {
if (m == j) {
j -= IP_RECORD_LENGTH;
m = j;
} else
j = m;
} else
return readLong3(m + 4);
}
// 如果循环结束了，那么i和j必定是相等的，这个记录为最可能的记录，但是并非
// 肯定就是，还要检查一下，如果是，就返回结束地址区的绝对偏移
m = readLong3(m + 4);
readIP(m, b4);
r = compareIP(ip, b4);
if (r <= 0)
return m;
else
return -1;
}
/**
* 得到begin偏移和end偏移中间位置记录的偏移
*
* @param begin
* @param end
* @return
*/
private long getMiddleOffset(long begin, long end) {
long records = (end - begin) / IP_RECORD_LENGTH;
records >>= 1;
if (records == 0)
records = 1;
return begin + records * IP_RECORD_LENGTH;
}
/**
* 给定一个ip国家地区记录的偏移，返回一个IPLocation结构
*
* @param offset
* @return
*/
private IPLocation getIPLocation(long offset) {
try {
// 跳过4字节ip
ipFile.seek(offset + 4);
// 读取第一个字节判断是否标志字节
byte b = ipFile.readByte();
if (b == AREA_FOLLOWED) {
// 读取国家偏移
long countryOffset = readLong3();
// 跳转至偏移处
ipFile.seek(countryOffset);
// 再检查一次标志字节，因为这个时候这个地方仍然可能是个重定向
b = ipFile.readByte();
if (b == NO_AREA) {
loc.country = readString(readLong3());
ipFile.seek(countryOffset + 4);
} else
loc.country = readString(countryOffset);
// 读取地区标志
loc.area = readArea(ipFile.getFilePointer());
} else if (b == NO_AREA) {
loc.country = readString(readLong3());
loc.area = readArea(offset + 8);
} else {
loc.country = readString(ipFile.getFilePointer() - 1);
loc.area = readArea(ipFile.getFilePointer());
}
return loc;
} catch (IOException e) {
return null;
}
}
/**
* @param offset
* @return
*/
private IPLocation getIPLocation(int offset) {
// 跳过4字节ip
mbb.position(offset + 4);
// 读取第一个字节判断是否标志字节
byte b = mbb.get();
if (b == AREA_FOLLOWED) {
// 读取国家偏移
int countryOffset = readInt3();
// 跳转至偏移处
mbb.position(countryOffset);
// 再检查一次标志字节，因为这个时候这个地方仍然可能是个重定向
b = mbb.get();
if (b == NO_AREA) {
loc.country = readString(readInt3());
mbb.position(countryOffset + 4);
} else
loc.country = readString(countryOffset);
// 读取地区标志
loc.area = readArea(mbb.position());
} else if (b == NO_AREA) {
loc.country = readString(readInt3());
loc.area = readArea(offset + 8);
} else {
loc.country = readString(mbb.position() - 1);
loc.area = readArea(mbb.position());
}
return loc;
}
/**
* 从offset偏移开始解析后面的字节，读出一个地区名
*
* @param offset
* @return 地区名字符串
* @throws IOException
*/
private String readArea(long offset) throws IOException {
ipFile.seek(offset);
byte b = ipFile.readByte();
if (b == 0x01 || b == 0x02) {
long areaOffset = readLong3(offset + 1);
if (areaOffset == 0)
return "未知地区";
else
return readString(areaOffset);
} else
return readString(offset);
}
/**
* @param offset
* @return
*/
private String readArea(int offset) {
mbb.position(offset);
byte b = mbb.get();
if (b == 0x01 || b == 0x02) {
int areaOffset = readInt3();
if (areaOffset == 0)
return "未知地区";
else
return readString(areaOffset);
} else
return readString(offset);
}
/**
* 从offset偏移处读取一个以0结束的字符串
*
* @param offset
* @return 读取的字符串，出错返回空字符串
*/
private String readString(long offset) {
try {
ipFile.seek(offset);
int i;
for (i = 0, buf[i] = ipFile.readByte(); buf[i] != 0; buf[++i] = ipFile.readByte())
;
if (i != 0)
return IPSeekerUtils.getString(buf, 0, i, "GBK");
} catch (IOException e) {
System.out.println(e.getMessage());
}
return "";
}
/**
* 从内存映射文件的offset位置得到一个0结尾字符串
*
* @param offset
* @return
*/
private String readString(int offset) {
try {
mbb.position(offset);
int i;
for (i = 0, buf[i] = mbb.get(); buf[i] != 0; buf[++i] = mbb.get())
;
if (i != 0)
return IPSeekerUtils.getString(buf, 0, i, "GBK");
} catch (IllegalArgumentException e) {
System.out.println(e.getMessage());
}
return "";
}
public String getAddress(String ip) {
String country = getCountry(ip).equals(" CZ88.NET") ? "" : getCountry(ip);
String area = getArea(ip).equals(" CZ88.NET") ? "" : getArea(ip);
String address = country + " " + area;
return address.trim();
}
/**
* * 用来封装ip相关信息，目前只有两个字段，ip所在的国家和地区
*
*
* @author swallow
*/
public class IPLocation {
public String country;
public String area;
public IPLocation() {
country = area = "";
}
public IPLocation getCopy() {
IPLocation ret = new IPLocation();
ret.country = country;
ret.area = area;
return ret;
}
}
/**
* 一条IP范围记录，不仅包括国家和区域，也包括起始IP和结束IP *
*
*
* @author gerry liu
*/
public class IPEntry {
public String beginIp;
public String endIp;
public String country;
public String area;
public IPEntry() {
beginIp = endIp = country = area = "";
}
public String toString() {
return this.area + " " + this.country + "IP Χ:" + this.beginIp + "-" + this.endIp;
}
}
/**
* 操作工具类
*
* @author gerryliu
*
*/
public static class IPSeekerUtils {
/**
* 从ip的字符串形式得到字节数组形式
*
* @param ip
* 字符串形式的ip
* @return 字节数组形式的ip
*/
public static byte[] getIpByteArrayFromString(String ip) {
byte[] ret = new byte[4];
java.util.StringTokenizer st = new java.util.StringTokenizer(ip, ".");
try {
ret[0] = (byte) (Integer.parseInt(st.nextToken()) & 0xFF);
ret[1] = (byte) (Integer.parseInt(st.nextToken()) & 0xFF);
ret[2] = (byte) (Integer.parseInt(st.nextToken()) & 0xFF);
ret[3] = (byte) (Integer.parseInt(st.nextToken()) & 0xFF);
} catch (Exception e) {
System.out.println(e.getMessage());
}
return ret;
}
/**
* 对原始字符串进行编码转换，如果失败，返回原始的字符串
*
* @param s
* 原始字符串
* @param srcEncoding
* 源编码方式
* @param destEncoding
* 目标编码方式
* @return 转换编码后的字符串，失败返回原始字符串
*/
public static String getString(String s, String srcEncoding, String destEncoding) {
try {
return new String(s.getBytes(srcEncoding), destEncoding);
} catch (UnsupportedEncodingException e) {
return s;
}
}
/**
* 根据某种编码方式将字节数组转换成字符串
*
* @param b
* 字节数组
* @param encoding
* 编码方式
* @return 如果encoding不支持，返回一个缺省编码的字符串
*/
public static String getString(byte[] b, String encoding) {
try {
return new String(b, encoding);
} catch (UnsupportedEncodingException e) {
return new String(b);
}
}
/**
* 根据某种编码方式将字节数组转换成字符串
*
* @param b
* 字节数组
* @param offset
* 要转换的起始位置
* @param len
* 要转换的长度
* @param encoding
* 编码方式
* @return 如果encoding不支持，返回一个缺省编码的字符串
*/
public static String getString(byte[] b, int offset, int len, String encoding) {
try {
return new String(b, offset, len, encoding);
} catch (UnsupportedEncodingException e) {
return new String(b, offset, len);
}
}
/**
* @param ip
* ip的字节数组形式
* @return 字符串形式的ip
*/
public static String getIpStringFromBytes(byte[] ip) {
StringBuffer sb = new StringBuffer();
sb.append(ip[0] & 0xFF);
sb.append('.');
sb.append(ip[1] & 0xFF);
sb.append('.');
sb.append(ip[2] & 0xFF);
sb.append('.');
sb.append(ip[3] & 0xFF);
return sb.toString();
}
}
/**
* 获取全部ip地址集合列表
*
* @return
*/
public List<String> getAllIp() {
List<String> list = new ArrayList<String>();
byte[] buf = new byte[4];
for (long i = ipBegin; i < ipEnd; i += IP_RECORD_LENGTH) {
try {
this.readIP(this.readLong3(i + 4), buf); // 读取ip，最终ip放到buf中
String ip = IPSeekerUtils.getIpStringFromBytes(buf);
list.add(ip);
} catch (Exception e) {
// nothing
}
}
return list;
}
}3.5.LogParser.java
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.HashMap;
import java.util.Map;
public class LogParser {
private Logger logger = LoggerFactory.getLogger(LogParser.class);
public Map<String, String> parse2(String log) {
Map<String, String> logInfo = new HashMap<String,String>();
IPParser ipParse = IPParser.getInstance();
if(StringUtils.isNotBlank(log)) {
String[] splits = log.split("\t");
String ip = splits[0];
String url = splits[1];
String sessionId = splits[2];
String time = splits[3];
String country = splits[4];
String province = splits[5];
String city = splits[6];
logInfo.put("ip",ip);
logInfo.put("url",url);
logInfo.put("sessionId",sessionId);
logInfo.put("time",time);
logInfo.put("country",country);
logInfo.put("province",province);
logInfo.put("city",city);
} else{
logger.error("日志记录的格式不正确：" + log);
}
return logInfo;
}
public Map<String, String> parse(String log) {
Map<String, String> logInfo = new HashMap<String,String>();
IPParser ipParse = IPParser.getInstance();
if(StringUtils.isNotBlank(log)) {
String[] splits = log.split("\001");
String ip = splits[13];
String url = splits[1];
String sessionId = splits[10];
String time = splits[17];
logInfo.put("ip",ip);
logInfo.put("url",url);
logInfo.put("sessionId",sessionId);
logInfo.put("time",time);
IPParser.RegionInfo regionInfo = ipParse.analyseIp(ip);
logInfo.put("country",regionInfo.getCountry());
logInfo.put("province",regionInfo.getProvince());
logInfo.put("city",regionInfo.getCity());
} else{
logger.error("日志记录的格式不正确：" + log);
}
return logInfo;
}
}3.6.PVStartAPP.java
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
/**
* 浏览量的统计
*/
public class PVStartApp {
public static void main(String[] args) throws Exception{
// driver类 八股文
Configuration configuration =new Configuration();
FileSystem fileSystem=FileSystem.get(configuration);
Path outputPath=new Path(args[1]);
if(fileSystem.exists(outputPath)){
fileSystem.delete(outputPath,true);
}
Job job =Job.getInstance(configuration);
job.setJarByClass(PVStartApp.class);
job.setMapperClass(Mymapper.class);
job.setReducerClass(MyReducer.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(LongWritable.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(LongWritable.class);
FileInputFormat.setInputPaths(job,new Path(args[0]));
FileOutputFormat.setOutputPath(job,new Path(args[1]));
job.waitForCompletion(true);
}
//Map
static class Mymapper extends Mapper<LongWritable, Text, Text, LongWritable> {
private Text KEY=new Text("key");
private LongWritable ONE=new LongWritable(1);
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
context.write(KEY,ONE);
}
}
//Reduce
static class MyReducer extends Reducer<Text,LongWritable, NullWritable,LongWritable>{
@Override
protected void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
long count =0;
for(LongWritable value :values){
count++;
}
context.write(NullWritable.get(),new LongWritable(count));
}
}
}
4.结果
最终结果到项目目录 执行 mvn clean package -DiskTest 打包 把打好的包上传到服务器
的～/lib下。然后把ip解析文件传输到服务器目录自己定，原始文件上传到服务器HDFS，之后在服务器中查看。
457
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
