11G数据的hive初测试

最新推荐文章于 2024-08-08 19:58:24 发布

iteye_4537

最新推荐文章于 2024-08-08 19:58:24 发布

阅读量114

点赞数

文章标签：大数据

首先，看到标题，我想就会有人喷我，①11G对于hadoop而言不值一提②做耗时测试本没有必要，毕竟hadoop用于线下系统。

我的想法：没那么复杂，只是想看下hadoop的文件处理能力和表现出的问题，只是单纯的享受这个解决问题并且得到结果的快感。

不那么多废话啦!直接进入正题：

1、数据来源：

apache的日志，接近900万条数据，大小为11个G。

2、创建表，并用正则匹配的方式导入数据，(这里hive的正则和普通正则，路由不同，之后会发新的博客讨论这个问题：(hive的正则)http://blog.csdn.net/lengzijian/article/details/7048575)：

这里正则值提取的日志当中的：site：当前站点，get：当前站点请求页，fore：之前站点------主要用于点击流，淡然是简单版的。为了以后应付更大文件做准备。

CREATE TABLE click (site STRING, get STRING, fore STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = ".*?- ([^ ]*) [GET]*[POST]* (/.*?) .*WEBTRENDS_ID=.* [http://]*(.+) .*","output.format.string" = "%1$s,%3$s,%2$s")STORED AS TEXTFILE;
3、导入数据，我这里有23个数据，分别代表一天的时间(可能少了一个小时)，可以把这23个数据连接在一起导入，但是我没有这样做，因为分别看每一个的时间，很容易的看到每个时间段的访问量，对于绘制监控图比较方便。

LOAD DATA LOCAL INPATH "/data/log/sdc_20110926_hr【00-22】.log" INTO TABLE click;

4、查询：因为刚刚做完没有经验，直接运行了

select count(*) from click;
出现以下信息：

Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201112071106_0005, Tracking URL = http://hadoop149:50030/jobdetails.jsp?jobid=job_201112071106_0005 Kill Command = /root/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=hdfs://192.168.30.149:9004 -kill job_201112071106_0005 2011-12-07 17:03:55,880 Stage-1 map = 0%, reduce = 0% 2011-12-07 17:04:04,936 Stage-1 map = 1%, reduce = 0% 2011-12-07 17:04:15,040 Stage-1 map = 2%, reduce = 0% 2011-12-07 17:04:17,057 Stage-1 map = 3%, reduce = 0% 2011-12-07 17:04:24,103 Stage-1 map = 4%, reduce = 0% 2011-12-07 17:04:26,121 Stage-1 map = 5%, reduce = 0% 2011-12-07 17:04:36,185 Stage-1 map = 6%, reduce = 0% 2011-12-07 17:04:38,199 Stage-1 map = 7%, reduce = 0% 2011-12-07 17:04:44,239 Stage-1 map = 7%, reduce = 2% 2011-12-07 17:04:50,275 Stage-1 map = 9%, reduce = 2% 2011-12-07 17:04:59,333 Stage-1 map = 11%, reduce = 3% 2011-12-07 17:05:08,386 Stage-1 map = 13%, reduce = 3% 2011-12-07 17:05:14,425 Stage-1 map = 13%, reduce = 4% 2011-12-07 17:05:17,450 Stage-1 map = 14%, reduce = 4% 2011-12-07 17:05:21,476 Stage-1 map = 15%, reduce = 4% 2011-12-07 17:05:25,505 Stage-1 map = 16%, reduce = 4% 2011-12-07 17:05:28,530 Stage-1 map = 16%, reduce = 5% 2011-12-07 17:05:34,568 Stage-1 map = 17%, reduce = 5% 2011-12-07 17:05:37,588 Stage-1 map = 18%, reduce = 5% 2011-12-07 17:05:43,629 Stage-1 map = 19%, reduce = 6% 2011-12-07 17:05:46,648 Stage-1 map = 20%, reduce = 6% 2011-12-07 17:05:55,705 Stage-1 map = 22%, reduce = 6% 2011-12-07 17:05:58,724 Stage-1 map = 22%, reduce = 7% 2011-12-07 17:06:04,761 Stage-1 map = 23%, reduce = 7% 2011-12-07 17:06:07,783 Stage-1 map = 24%, reduce = 7% 2011-12-07 17:06:16,839 Stage-1 map = 25%, reduce = 8% 2011-12-07 17:06:18,853 Stage-1 map = 26%, reduce = 8% 2011-12-07 17:06:25,900 Stage-1 map = 27%, reduce = 8% 2011-12-07 17:06:31,937 Stage-1 map = 27%, reduce = 9% 2011-12-07 17:06:33,951 Stage-1 map = 28%, reduce = 9% 2011-12-07 17:06:36,973 Stage-1 map = 29%, reduce = 9% 2011-12-07 17:06:43,010 Stage-1 map = 30%, reduce = 9% 2011-12-07 17:06:47,039 Stage-1 map = 31%, reduce = 10% 2011-12-07 17:06:55,088 Stage-1 map = 32%, reduce = 10% 2011-12-07 17:06:56,095 Stage-1 map = 33%, reduce = 10% 2011-12-07 17:07:03,140 Stage-1 map = 34%, reduce = 10% 2011-12-07 17:07:08,172 Stage-1 map = 35%, reduce = 11% 2011-12-07 17:07:14,210 Stage-1 map = 36%, reduce = 11% 2011-12-07 17:07:18,241 Stage-1 map = 37%, reduce = 11% 2011-12-07 17:07:23,275 Stage-1 map = 38%, reduce = 12% 2011-12-07 17:07:29,312 Stage-1 map = 39%, reduce = 12% 2011-12-07 17:07:32,331 Stage-1 map = 40%, reduce = 13% 2011-12-07 17:07:38,373 Stage-1 map = 41%, reduce = 13% 2011-12-07 17:07:41,392 Stage-1 map = 42%, reduce = 13% 2011-12-07 17:07:47,429 Stage-1 map = 42%, reduce = 14% 2011-12-07 17:07:48,436 Stage-1 map = 43%, reduce = 14% 2011-12-07 17:07:51,456 Stage-1 map = 44%, reduce = 14% 2011-12-07 17:07:56,497 Stage-1 map = 45%, reduce = 14% 2011-12-07 17:08:02,533 Stage-1 map = 46%, reduce = 14% 2011-12-07 17:08:08,568 Stage-1 map = 46%, reduce = 15% 2011-12-07 17:08:09,575 Stage-1 map = 47%, reduce = 15% 2011-12-07 17:08:12,594 Stage-1 map = 48%, reduce = 15% 2011-12-07 17:08:18,634 Stage-1 map = 49%, reduce = 15% 2011-12-07 17:08:23,664 Stage-1 map = 50%, reduce = 16% 2011-12-07 17:08:29,701 Stage-1 map = 51%, reduce = 16% 2011-12-07 17:08:32,718 Stage-1 map = 51%, reduce = 17% 2011-12-07 17:08:35,736 Stage-1 map = 52%, reduce = 17% 2011-12-07 17:08:39,761 Stage-1 map = 53%, reduce = 17% 2011-12-07 17:08:44,791 Stage-1 map = 54%, reduce = 17% 2011-12-07 17:08:51,832 Stage-1 map = 55%, reduce = 17% 2011-12-07 17:08:53,845 Stage-1 map = 55%, reduce = 18% 2011-12-07 17:08:56,864 Stage-1 map = 56%, reduce = 18% 2011-12-07 17:09:00,887 Stage-1 map = 57%, reduce = 18% 2011-12-07 17:09:05,920 Stage-1 map = 58%, reduce = 18% 2011-12-07 17:09:08,939 Stage-1 map = 58%, reduce = 19% 2011-12-07 17:09:14,974 Stage-1 map = 60%, reduce = 19% 2011-12-07 17:09:24,029 Stage-1 map = 61%, reduce = 20% 2011-12-07 17:09:27,049 Stage-1 map = 62%, reduce = 20% 2011-12-07 17:09:33,086 Stage-1 map = 63%, reduce = 20% 2011-12-07 17:09:36,105 Stage-1 map = 64%, reduce = 20% 2011-12-07 17:09:39,124 Stage-1 map = 64%, reduce = 21% 2011-12-07 17:09:45,160 Stage-1 map = 65%, reduce = 21% 2011-12-07 17:09:46,167 Stage-1 map = 66%, reduce = 21% 2011-12-07 17:09:54,216 Stage-1 map = 67%, reduce = 22% 2011-12-07 17:09:58,241 Stage-1 map = 68%, reduce = 22% 2011-12-07 17:10:03,272 Stage-1 map = 69%, reduce = 22% 2011-12-07 17:10:09,309 Stage-1 map = 69%, reduce = 23% 2011-12-07 17:10:10,316 Stage-1 map = 70%, reduce = 23% 2011-12-07 17:10:15,345 Stage-1 map = 71%, reduce = 23% 2011-12-07 17:10:21,382 Stage-1 map = 72%, reduce = 23% 2011-12-07 17:10:24,401 Stage-1 map = 73%, reduce = 24% 2011-12-07 17:10:30,438 Stage-1 map = 74%, reduce = 24% 2011-12-07 17:10:33,457 Stage-1 map = 75%, reduce = 24% 2011-12-07 17:10:39,497 Stage-1 map = 76%, reduce = 24% 2011-12-07 17:10:42,517 Stage-1 map = 77%, reduce = 24% 2011-12-07 17:10:45,537 Stage-1 map = 77%, reduce = 25% 2011-12-07 17:10:51,589 Stage-1 map = 79%, reduce = 25% 2011-12-07 17:11:00,645 Stage-1 map = 80%, reduce = 26% 2011-12-07 17:11:06,683 Stage-1 map = 81%, reduce = 27% 2011-12-07 17:11:09,700 Stage-1 map = 82%, reduce = 27% 2011-12-07 17:11:18,755 Stage-1 map = 83%, reduce = 27% 2011-12-07 17:11:21,774 Stage-1 map = 84%, reduce = 27% 2011-12-07 17:11:30,854 Stage-1 map = 85%, reduce = 28% 2011-12-07 17:11:33,874 Stage-1 map = 86%, reduce = 28% 2011-12-07 17:11:39,912 Stage-1 map = 87%, reduce = 28% 2011-12-07 17:11:45,949 Stage-1 map = 88%, reduce = 29% 2011-12-07 17:11:48,970 Stage-1 map = 89%, reduce = 29% 2011-12-07 17:11:55,008 Stage-1 map = 90%, reduce = 29% 2011-12-07 17:11:58,028 Stage-1 map = 91%, reduce = 29% 2011-12-07 17:12:01,048 Stage-1 map = 91%, reduce = 30% 2011-12-07 17:12:04,067 Stage-1 map = 92%, reduce = 30% 2011-12-07 17:12:07,088 Stage-1 map = 93%, reduce = 30% 2011-12-07 17:12:13,127 Stage-1 map = 94%, reduce = 31% 2011-12-07 17:12:19,167 Stage-1 map = 95%, reduce = 31% 2011-12-07 17:12:22,186 Stage-1 map = 96%, reduce = 31% 2011-12-07 17:12:25,205 Stage-1 map = 97%, reduce = 31% 2011-12-07 17:12:28,225 Stage-1 map = 98%, reduce = 31% 2011-12-07 17:12:31,251 Stage-1 map = 99%, reduce = 31% 2011-12-07 17:12:34,275 Stage-1 map = 100%, reduce = 33% 2011-12-07 17:12:40,312 Stage-1 map = 100%, reduce = 100% Ended Job = job_201112071106_0005 OK 8625654 Time taken: 541.638 seconds在执行过程中，一台电脑(子节点)一直再响，看了下cpu，双核几乎全被占满。最后结果运行了