Hive-on-MR与Hive-on-Tez的性能比较

最新推荐文章于 2022-07-07 11:24:50 发布
最新推荐文章于 2022-07-07 11:24:50 发布
阅读量2k 收藏 1
点赞数
文章标签： 大数据 java
原文链接：https://my.oschina.net/whulyx/blog/731940
版权

    实验数据：10G大小的TPCH数据（8个表）；

    实验环境：1Master节点+8Slave节点，每个均为4core，4G内存，30G磁盘，磁盘读写速度50M/s；

    ①单个查询之q1_pricing_summary_report.hive

SELECT 
  L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1) 
FROM 
  lineitem 
WHERE 
  L_SHIPDATE<='1998-09-02' 
GROUP BY L_RETURNFLAG, L_LINESTATUS 
ORDER BY L_RETURNFLAG, L_LINESTATUS;

    ——set hive.execution.engine=mr

hive> SELECT 
    >   L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1) 
    > FROM 
    >   lineitem 
    > WHERE 
    >   L_SHIPDATE<='1998-09-02' 
    > GROUP BY L_RETURNFLAG, L_LINESTATUS 
    > ORDER BY L_RETURNFLAG, L_LINESTATUS;
16/08/12 18:19:57 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = lwh_20160812181955_0ba24dd7-25a8-44c6-9005-f5adacfc1b08
Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 31
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 18:20:02 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
16/08/12 18:20:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 18:20:04 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 18:20:04 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 18:21:48 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 18:21:48 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
16/08/12 18:21:49 INFO mapreduce.JobSubmitter: number of splits:29
16/08/12 18:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0007
16/08/12 18:21:50 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0007
16/08/12 18:21:50 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0007/
Starting Job = job_1470994648796_0007, Tracking URL = http://node643:8088/proxy/application_1470994648796_0007/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0007
Hadoop job information for Stage-1: number of mappers: 29; number of reducers: 31
16/08/12 18:22:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 18:22:15,564 Stage-1 map = 0%,  reduce = 0%
2016-08-12 18:22:40,147 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 40.74 sec
2016-08-12 18:22:45,548 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 189.74 sec
2016-08-12 18:22:46,625 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 211.75 sec
2016-08-12 18:22:50,940 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 298.17 sec
2016-08-12 18:22:52,019 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 306.72 sec
2016-08-12 18:22:53,097 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU 313.03 sec
2016-08-12 18:22:54,173 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 341.25 sec
2016-08-12 18:22:58,499 Stage-1 map = 20%,  reduce = 0%, Cumulative CPU 408.43 sec
2016-08-12 18:23:00,651 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 458.76 sec
2016-08-12 18:23:01,724 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU 463.35 sec
2016-08-12 18:23:03,873 Stage-1 map = 47%,  reduce = 0%, Cumulative CPU 474.89 sec
2016-08-12 18:23:09,249 Stage-1 map = 48%,  reduce = 0%, Cumulative CPU 485.99 sec
2016-08-12 18:23:11,405 Stage-1 map = 51%,  reduce = 0%, Cumulative CPU 492.84 sec
2016-08-12 18:23:12,498 Stage-1 map = 53%,  reduce = 0%, Cumulative CPU 509.8 sec
2016-08-12 18:23:14,676 Stage-1 map = 53%,  reduce = 1%, Cumulative CPU 533.6 sec
2016-08-12 18:23:16,828 Stage-1 map = 55%,  reduce = 1%, Cumulative CPU 598.25 sec
2016-08-12 18:23:20,048 Stage-1 map = 55%,  reduce = 2%, Cumulative CPU 636.57 sec
2016-08-12 18:23:21,134 Stage-1 map = 56%,  reduce = 2%, Cumulative CPU 649.37 sec
2016-08-12 18:23:23,298 Stage-1 map = 57%,  reduce = 2%, Cumulative CPU 669.42 sec
2016-08-12 18:23:25,443 Stage-1 map = 61%,  reduce = 2%, Cumulative CPU 689.89 sec
2016-08-12 18:23:26,531 Stage-1 map = 62%,  reduce = 2%, Cumulative CPU 697.35 sec
2016-08-12 18:23:27,602 Stage-1 map = 64%,  reduce = 2%, Cumulative CPU 715.54 sec
2016-08-12 18:23:28,671 Stage-1 map = 67%,  reduce = 2%, Cumulative CPU 728.59 sec
2016-08-12 18:23:31,881 Stage-1 map = 74%,  reduce = 2%, Cumulative CPU 777.91 sec
2016-08-12 18:23:32,951 Stage-1 map = 83%,  reduce = 3%, Cumulative CPU 791.75 sec
2016-08-12 18:23:36,158 Stage-1 map = 83%,  reduce = 4%, Cumulative CPU 809.7 sec
2016-08-12 18:23:39,366 Stage-1 map = 84%,  reduce = 4%, Cumulative CPU 819.73 sec
2016-08-12 18:23:40,450 Stage-1 map = 85%,  reduce = 4%, Cumulative CPU 822.8 sec
2016-08-12 18:23:42,587 Stage-1 map = 86%,  reduce = 4%, Cumulative CPU 830.3 sec
2016-08-12 18:23:45,806 Stage-1 map = 89%,  reduce = 4%, Cumulative CPU 865.81 sec
2016-08-12 18:23:46,880 Stage-1 map = 89%,  reduce = 7%, Cumulative CPU 872.51 sec
2016-08-12 18:23:47,951 Stage-1 map = 89%,  reduce = 8%, Cumulative CPU 883.72 sec
2016-08-12 18:23:49,021 Stage-1 map = 91%,  reduce = 9%, Cumulative CPU 891.21 sec
2016-08-12 18:23:50,097 Stage-1 map = 93%,  reduce = 9%, Cumulative CPU 893.01 sec
2016-08-12 18:23:52,247 Stage-1 map = 93%,  reduce = 10%, Cumulative CPU 900.14 sec
2016-08-12 18:23:53,317 Stage-1 map = 95%,  reduce = 10%, Cumulative CPU 908.03 sec
2016-08-12 18:23:57,624 Stage-1 map = 95%,  reduce = 11%, Cumulative CPU 916.72 sec
2016-08-12 18:24:00,843 Stage-1 map = 100%,  reduce = 11%, Cumulative CPU 926.87 sec
2016-08-12 18:24:01,916 Stage-1 map = 100%,  reduce = 14%, Cumulative CPU 930.05 sec
2016-08-12 18:24:02,988 Stage-1 map = 100%,  reduce = 21%, Cumulative CPU 942.11 sec
2016-08-12 18:24:04,062 Stage-1 map = 100%,  reduce = 30%, Cumulative CPU 962.53 sec
2016-08-12 18:24:05,135 Stage-1 map = 100%,  reduce = 39%, Cumulative CPU 993.55 sec
2016-08-12 18:24:07,282 Stage-1 map = 100%,  reduce = 42%, Cumulative CPU 1001.0 sec
2016-08-12 18:24:11,593 Stage-1 map = 100%,  reduce = 48%, Cumulative CPU 1015.47 sec
2016-08-12 18:24:13,733 Stage-1 map = 100%,  reduce = 52%, Cumulative CPU 1020.94 sec
2016-08-12 18:24:14,800 Stage-1 map = 100%,  reduce = 65%, Cumulative CPU 1047.42 sec
2016-08-12 18:24:15,868 Stage-1 map = 100%,  reduce = 77%, Cumulative CPU 1073.01 sec
2016-08-12 18:24:16,936 Stage-1 map = 100%,  reduce = 84%, Cumulative CPU 1083.68 sec
2016-08-12 18:24:20,139 Stage-1 map = 100%,  reduce = 87%, Cumulative CPU 1088.29 sec
2016-08-12 18:24:21,205 Stage-1 map = 100%,  reduce = 94%, Cumulative CPU 1100.29 sec
2016-08-12 18:24:23,354 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 1112.48 sec
MapReduce Total cumulative CPU time: 18 minutes 32 seconds 480 msec
Ended Job = job_1470994648796_0007
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 18:24:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 18:24:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 18:24:28 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 18:26:10 INFO input.FileInputFormat: Total input paths to process : 31
16/08/12 18:26:10 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
16/08/12 18:26:11 INFO mapreduce.JobSubmitter: number of splits:6
16/08/12 18:26:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0008
16/08/12 18:26:12 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0008
16/08/12 18:26:12 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0008/
Starting Job = job_1470994648796_0008, Tracking URL = http://node643:8088/proxy/application_1470994648796_0008/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0008
Hadoop job information for Stage-2: number of mappers: 6; number of reducers: 1
16/08/12 18:26:35 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 18:26:35,127 Stage-2 map = 0%,  reduce = 0%
2016-08-12 18:26:58,048 Stage-2 map = 50%,  reduce = 0%, Cumulative CPU 11.12 sec
2016-08-12 18:27:00,187 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 21.08 sec
2016-08-12 18:27:07,672 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 26.1 sec
MapReduce Total cumulative CPU time: 26 seconds 100 msec
Ended Job = job_1470994648796_0008
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 29  Reduce: 31   Cumulative CPU: 1112.48 sec   HDFS Read: 7776522714 HDFS Write: 3305 SUCCESS
Stage-Stage-2: Map: 6  Reduce: 1   Cumulative CPU: 26.1 sec   HDFS Read: 39220 HDFS Write: 720 SUCCESS
Total MapReduce CPU Time Spent: 18 minutes 58 seconds 580 msec
OK
16/08/12 18:27:11 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 18:27:11 INFO mapred.FileInputFormat: Total input paths to process : 1
A       F       3.77518399E8    5.660657277972468E11    5.377591042780681E11    5.592766708921187E11    25.500975103007097      38237.15100895833       0.050006574540269355    14804077
N       F       9851614.0       1.4767438399169975E10   1.4028805792211395E10   1.459049099836673E10    25.522448302840946      38257.81066008108       0.04997336773765436     385998
N       O       7.43124873E8    1.1143022869018765E12   1.0585809221449628E12   1.100937000170583E12    25.498075870689316      38233.90292348169       0.0500008118209974      29144351
R       F       3.7773283E8     5.664310549759971E11    5.3811092266476355E11   5.59634780885086E11     25.50838478968014       38251.21927355956       0.04999679231411408     14808183
Time taken: 436.074 seconds, Fetched: 4 row(s)

    ——set hive.execution.engine=tez

hive> set hive.execution.engine=tez;
hive> SELECT 
    >   L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY), SUM(L_EXTENDEDPRICE), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)), SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)), AVG(L_QUANTITY), AVG(L_EXTENDEDPRICE), AVG(L_DISCOUNT), COUNT(1) 
    > FROM 
    >   lineitem 
    > WHERE 
    >   L_SHIPDATE<='1998-09-02' 
    > GROUP BY L_RETURNFLAG, L_LINESTATUS 
    > ORDER BY L_RETURNFLAG, L_LINESTATUS;
16/08/12 19:01:38 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/08/12 19:01:38 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
Query ID = lwh_20160812190132_2f9c1fad-5f40-4ad0-99bf-57a921acf812
Total jobs = 1
Launching Job 1 out of 1
16/08/12 19:01:39 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/08/12 19:03:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:03:03 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0009
16/08/12 19:03:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080


Status: Running (Executing on YARN cluster with App id application_1470994648796_0009)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED  
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED     58         58        0        0       0       0  
Reducer 2 ...... container     SUCCEEDED     11         11        0        0       0       0  
Reducer 3 ...... container     SUCCEEDED      1          1        0        0       0       0  
----------------------------------------------------------------------------------------------
VERTICES: 03/03  [==========================>>] 100%  ELAPSED TIME: 72.51 s    
----------------------------------------------------------------------------------------------
OK
16/08/12 19:04:42 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 19:04:42 INFO mapred.FileInputFormat: Total input paths to process : 1
A       F       3.77518399E8    5.660657277972496E11    5.3775910427806433E11   5.592766708921171E11    25.500975103007097      38237.15100895852       0.05000657454031599     14804077
N       F       9851614.0       1.4767438399169992E10   1.4028805792211395E10   1.4590490998366734E10   25.522448302840946      38257.81066008112       0.049973367737656114    385998
N       O       7.43124873E8    1.1143022869018794E12   1.0585809221449642E12   1.1009370001705938E12   25.498075870689316      38233.902923481786      0.050000811821159556    29144351
R       F       3.7773283E8     5.664310549760012E11    5.3811092266476697E11   5.596347808850865E11    25.50838478968014       38251.219273559844      0.04999679231416023     14808183
Time taken: 189.653 seconds, Fetched: 4 row(s)

    ②单个查询之q5_local_supplier_volume.hive

select 
  n_name, sum(l_extendedprice * (1 - l_discount)) as revenue 
from
  customer c join
    ( select n_name, l_extendedprice, l_discount, s_nationkey, o_custkey from orders o join
      ( select n_name, l_extendedprice, l_discount, l_orderkey, s_nationkey from lineitem l join
        ( select n_name, s_suppkey, s_nationkey from supplier s join
          ( select n_name, n_nationkey 
            from nation n join region r 
            on n.n_regionkey = r.r_regionkey and r.r_name = 'ASIA'
          ) n1 on s.s_nationkey = n1.n_nationkey
        ) s1 on l.l_suppkey = s1.s_suppkey
      ) l1 on l1.l_orderkey = o.o_orderkey and o.o_orderdate >= '1994-01-01' 
              and o.o_orderdate < '1995-01-01'
) o1 
on c.c_nationkey = o1.s_nationkey and c.c_custkey = o1.o_custkey
group by n_name 
order by revenue desc;

    ——set hive.execution.engine=mr

lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q5_local_supplier_volume.hive 
which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 19:20:21 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
16/08/12 19:20:25 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = lwh_20160812192025_499bf4fc-1185-495a-b02c-e81285fd7740
Total jobs = 15
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 19:20:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-08-12 19:20:46     Starting to launch local task to process map join;      maximum memory = 477626368
16/08/12 19:20:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 19:20:49 INFO mapred.FileInputFormat: Total input paths to process : 1
2016-08-12 19:20:49     Dump the side-table for tag: 1 with group count: 1 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile81--.hashtable
2016-08-12 19:20:49     Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10024/HashTable-Stage-22/MapJoin-mapfile81--.hashtable (278 bytes)
2016-08-12 19:20:49     End of local task; Time Taken: 3.023 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
16/08/12 19:20:51 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
16/08/12 19:20:52 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:20:53 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:20:53 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:22:38 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 19:22:38 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
16/08/12 19:22:39 INFO mapreduce.JobSubmitter: number of splits:1
16/08/12 19:22:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0011
16/08/12 19:22:40 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0011
16/08/12 19:22:40 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0011/
Starting Job = job_1470994648796_0011, Tracking URL = http://node643:8088/proxy/application_1470994648796_0011/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0011
Hadoop job information for Stage-22: number of mappers: 1; number of reducers: 0
16/08/12 19:23:09 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:23:09,546 Stage-22 map = 0%,  reduce = 0%
2016-08-12 19:23:37,474 Stage-22 map = 100%,  reduce = 0%, Cumulative CPU 6.64 sec
MapReduce Total cumulative CPU time: 6 seconds 640 msec
Ended Job = job_1470994648796_0011
Stage-29 is selected by condition resolver.
Stage-30 is filtered out by condition resolver.
Stage-7 is filtered out by condition resolver.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 19:23:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-08-12 19:23:52     Starting to launch local task to process map join;      maximum memory = 477626368
16/08/12 19:23:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 19:23:54 INFO mapred.FileInputFormat: Total input paths to process : 1
2016-08-12 19:23:54     Dump the side-table for tag: 1 with group count: 5 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10020/HashTable-Stage-19/MapJoin-mapfile61--.hashtable
2016-08-12 19:23:54     Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10020/HashTable-Stage-19/MapJoin-mapfile61--.hashtable (391 bytes)
2016-08-12 19:23:54     End of local task; Time Taken: 2.383 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 3 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
16/08/12 19:23:58 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:23:59 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:23:59 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:25:43 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 19:25:43 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
16/08/12 19:25:44 INFO mapreduce.JobSubmitter: number of splits:1
16/08/12 19:25:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0012
16/08/12 19:25:45 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0012
16/08/12 19:25:45 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0012/
Starting Job = job_1470994648796_0012, Tracking URL = http://node643:8088/proxy/application_1470994648796_0012/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0012
Hadoop job information for Stage-19: number of mappers: 1; number of reducers: 0
16/08/12 19:26:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:26:15,132 Stage-19 map = 0%,  reduce = 0%
2016-08-12 19:26:42,840 Stage-19 map = 100%,  reduce = 0%, Cumulative CPU 9.47 sec
MapReduce Total cumulative CPU time: 9 seconds 470 msec
Ended Job = job_1470994648796_0012
Stage-27 is selected by condition resolver.
Stage-28 is filtered out by condition resolver.
Stage-6 is filtered out by condition resolver.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 19:26:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-08-12 19:26:57     Starting to launch local task to process map join;      maximum memory = 477626368
16/08/12 19:26:59 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 19:26:59 INFO mapred.FileInputFormat: Total input paths to process : 1
2016-08-12 19:27:00     Dump the side-table for tag: 1 with group count: 20037 into file: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10016/HashTable-Stage-16/MapJoin-mapfile41--.hashtable
2016-08-12 19:27:00     Uploaded 1 File to: file:/home/lwh/hive/warehouse/ec5281d4-734d-498b-bd4e-138b399d7423/hive_2016-08-12_19-20-25_150_2659824356155076166-1/-local-10016/HashTable-Stage-16/MapJoin-mapfile41--.hashtable (594855 bytes)
2016-08-12 19:27:00     End of local task; Time Taken: 2.755 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 5 out of 15
Number of reduce tasks is set to 0 since there's no reduce operator
16/08/12 19:27:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:27:02 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:27:02 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:28:45 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 19:28:45 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
16/08/12 19:28:46 INFO mapreduce.JobSubmitter: number of splits:29
16/08/12 19:28:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0013
16/08/12 19:28:48 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0013
16/08/12 19:28:48 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0013/
Starting Job = job_1470994648796_0013, Tracking URL = http://node643:8088/proxy/application_1470994648796_0013/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0013
Hadoop job information for Stage-16: number of mappers: 29; number of reducers: 0
16/08/12 19:29:15 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:29:15,011 Stage-16 map = 0%,  reduce = 0%
2016-08-12 19:29:35,792 Stage-16 map = 2%,  reduce = 0%, Cumulative CPU 19.35 sec
2016-08-12 19:29:39,009 Stage-16 map = 3%,  reduce = 0%, Cumulative CPU 91.45 sec
2016-08-12 19:29:43,324 Stage-16 map = 7%,  reduce = 0%, Cumulative CPU 199.14 sec
2016-08-12 19:29:45,479 Stage-16 map = 14%,  reduce = 0%, Cumulative CPU 225.15 sec
2016-08-12 19:29:47,625 Stage-16 map = 16%,  reduce = 0%, Cumulative CPU 244.67 sec
2016-08-12 19:29:48,696 Stage-16 map = 22%,  reduce = 0%, Cumulative CPU 281.88 sec
2016-08-12 19:29:49,779 Stage-16 map = 31%,  reduce = 0%, Cumulative CPU 293.2 sec
2016-08-12 19:29:50,846 Stage-16 map = 34%,  reduce = 0%, Cumulative CPU 312.23 sec
2016-08-12 19:29:51,915 Stage-16 map = 36%,  reduce = 0%, Cumulative CPU 330.85 sec
2016-08-12 19:29:52,986 Stage-16 map = 40%,  reduce = 0%, Cumulative CPU 337.59 sec
2016-08-12 19:29:54,049 Stage-16 map = 41%,  reduce = 0%, Cumulative CPU 341.42 sec
2016-08-12 19:29:55,129 Stage-16 map = 43%,  reduce = 0%, Cumulative CPU 347.94 sec
2016-08-12 19:29:56,203 Stage-16 map = 45%,  reduce = 0%, Cumulative CPU 349.7 sec
2016-08-12 19:29:57,277 Stage-16 map = 47%,  reduce = 0%, Cumulative CPU 358.26 sec
2016-08-12 19:29:59,426 Stage-16 map = 48%,  reduce = 0%, Cumulative CPU 361.21 sec
2016-08-12 19:30:00,506 Stage-16 map = 52%,  reduce = 0%, Cumulative CPU 379.57 sec
2016-08-12 19:30:05,873 Stage-16 map = 53%,  reduce = 0%, Cumulative CPU 443.3 sec
2016-08-12 19:30:06,941 Stage-16 map = 57%,  reduce = 0%, Cumulative CPU 460.63 sec
2016-08-12 19:30:08,009 Stage-16 map = 60%,  reduce = 0%, Cumulative CPU 483.02 sec
2016-08-12 19:30:11,219 Stage-16 map = 64%,  reduce = 0%, Cumulative CPU 553.62 sec
2016-08-12 19:30:12,287 Stage-16 map = 67%,  reduce = 0%, Cumulative CPU 589.04 sec
2016-08-12 19:30:13,355 Stage-16 map = 72%,  reduce = 0%, Cumulative CPU 598.74 sec
2016-08-12 19:30:15,494 Stage-16 map = 74%,  reduce = 0%, Cumulative CPU 622.0 sec
2016-08-12 19:30:16,565 Stage-16 map = 78%,  reduce = 0%, Cumulative CPU 632.12 sec
2016-08-12 19:30:17,638 Stage-16 map = 81%,  reduce = 0%, Cumulative CPU 641.87 sec
2016-08-12 19:30:18,717 Stage-16 map = 83%,  reduce = 0%, Cumulative CPU 651.4 sec
2016-08-12 19:30:19,784 Stage-16 map = 84%,  reduce = 0%, Cumulative CPU 654.3 sec
2016-08-12 19:30:20,851 Stage-16 map = 90%,  reduce = 0%, Cumulative CPU 670.98 sec
2016-08-12 19:30:21,920 Stage-16 map = 91%,  reduce = 0%, Cumulative CPU 674.26 sec
2016-08-12 19:30:22,992 Stage-16 map = 93%,  reduce = 0%, Cumulative CPU 682.34 sec
2016-08-12 19:30:26,194 Stage-16 map = 98%,  reduce = 0%, Cumulative CPU 697.53 sec
2016-08-12 19:30:27,261 Stage-16 map = 100%,  reduce = 0%, Cumulative CPU 698.01 sec
MapReduce Total cumulative CPU time: 11 minutes 38 seconds 10 msec
Ended Job = job_1470994648796_0013
Stage-25 is filtered out by condition resolver.
Stage-26 is filtered out by condition resolver.
Stage-5 is selected by condition resolver.
Launching Job 6 out of 15
Number of reduce tasks not specified. Estimated from input data size: 10
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 19:30:31 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:30:31 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:30:31 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:32:16 INFO input.FileInputFormat: Total input paths to process : 30
16/08/12 19:32:16 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 4364567
16/08/12 19:32:16 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
16/08/12 19:32:17 INFO mapreduce.JobSubmitter: number of splits:15
16/08/12 19:32:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0014
16/08/12 19:32:18 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0014
16/08/12 19:32:18 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0014/
Starting Job = job_1470994648796_0014, Tracking URL = http://node643:8088/proxy/application_1470994648796_0014/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0014
Hadoop job information for Stage-5: number of mappers: 15; number of reducers: 10
16/08/12 19:32:47 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:32:47,002 Stage-5 map = 0%,  reduce = 0%
2016-08-12 19:33:05,006 Stage-5 map = 2%,  reduce = 0%, Cumulative CPU 16.74 sec
2016-08-12 19:33:08,223 Stage-5 map = 4%,  reduce = 0%, Cumulative CPU 20.43 sec
2016-08-12 19:33:09,293 Stage-5 map = 7%,  reduce = 0%, Cumulative CPU 33.74 sec
2016-08-12 19:33:14,659 Stage-5 map = 8%,  reduce = 0%, Cumulative CPU 140.57 sec
2016-08-12 19:33:15,734 Stage-5 map = 19%,  reduce = 0%, Cumulative CPU 157.44 sec
2016-08-12 19:33:16,804 Stage-5 map = 39%,  reduce = 0%, Cumulative CPU 236.11 sec
2016-08-12 19:33:17,874 Stage-5 map = 46%,  reduce = 0%, Cumulative CPU 245.0 sec
2016-08-12 19:33:20,023 Stage-5 map = 55%,  reduce = 0%, Cumulative CPU 271.72 sec
2016-08-12 19:33:21,092 Stage-5 map = 58%,  reduce = 0%, Cumulative CPU 278.26 sec
2016-08-12 19:33:22,172 Stage-5 map = 68%,  reduce = 1%, Cumulative CPU 293.54 sec
2016-08-12 19:33:23,248 Stage-5 map = 73%,  reduce = 1%, Cumulative CPU 302.53 sec
2016-08-12 19:33:25,403 Stage-5 map = 76%,  reduce = 2%, Cumulative CPU 320.42 sec
2016-08-12 19:33:26,477 Stage-5 map = 79%,  reduce = 2%, Cumulative CPU 330.54 sec
2016-08-12 19:33:28,621 Stage-5 map = 81%,  reduce = 2%, Cumulative CPU 346.81 sec
2016-08-12 19:33:29,690 Stage-5 map = 84%,  reduce = 7%, Cumulative CPU 356.93 sec
2016-08-12 19:33:31,840 Stage-5 map = 85%,  reduce = 7%, Cumulative CPU 372.83 sec
2016-08-12 19:33:32,906 Stage-5 map = 86%,  reduce = 8%, Cumulative CPU 377.17 sec
2016-08-12 19:33:35,047 Stage-5 map = 86%,  reduce = 10%, Cumulative CPU 385.15 sec
2016-08-12 19:33:36,106 Stage-5 map = 86%,  reduce = 14%, Cumulative CPU 390.71 sec
2016-08-12 19:33:37,174 Stage-5 map = 89%,  reduce = 16%, Cumulative CPU 401.5 sec
2016-08-12 19:33:38,252 Stage-5 map = 93%,  reduce = 16%, Cumulative CPU 417.24 sec
2016-08-12 19:33:39,325 Stage-5 map = 93%,  reduce = 18%, Cumulative CPU 422.06 sec
2016-08-12 19:33:40,395 Stage-5 map = 93%,  reduce = 20%, Cumulative CPU 429.3 sec
2016-08-12 19:33:41,466 Stage-5 map = 93%,  reduce = 21%, Cumulative CPU 430.34 sec
2016-08-12 19:33:44,689 Stage-5 map = 93%,  reduce = 24%, Cumulative CPU 445.4 sec
2016-08-12 19:33:48,979 Stage-5 map = 93%,  reduce = 27%, Cumulative CPU 453.62 sec
2016-08-12 19:34:00,732 Stage-5 map = 96%,  reduce = 27%, Cumulative CPU 481.33 sec
2016-08-12 19:34:01,806 Stage-5 map = 98%,  reduce = 27%, Cumulative CPU 496.84 sec
2016-08-12 19:34:02,873 Stage-5 map = 98%,  reduce = 29%, Cumulative CPU 498.16 sec
2016-08-12 19:34:03,940 Stage-5 map = 98%,  reduce = 30%, Cumulative CPU 498.71 sec
2016-08-12 19:34:05,006 Stage-5 map = 98%,  reduce = 31%, Cumulative CPU 499.47 sec
2016-08-12 19:34:17,843 Stage-5 map = 100%,  reduce = 31%, Cumulative CPU 526.06 sec
2016-08-12 19:34:18,917 Stage-5 map = 100%,  reduce = 32%, Cumulative CPU 528.93 sec
2016-08-12 19:34:19,987 Stage-5 map = 100%,  reduce = 33%, Cumulative CPU 532.83 sec
2016-08-12 19:34:21,054 Stage-5 map = 100%,  reduce = 37%, Cumulative CPU 552.13 sec
2016-08-12 19:34:22,123 Stage-5 map = 100%,  reduce = 39%, Cumulative CPU 558.94 sec
2016-08-12 19:34:23,194 Stage-5 map = 100%,  reduce = 41%, Cumulative CPU 565.28 sec
2016-08-12 19:34:24,272 Stage-5 map = 100%,  reduce = 46%, Cumulative CPU 583.96 sec
2016-08-12 19:34:25,341 Stage-5 map = 100%,  reduce = 47%, Cumulative CPU 590.27 sec
2016-08-12 19:34:26,408 Stage-5 map = 100%,  reduce = 50%, Cumulative CPU 599.89 sec
2016-08-12 19:34:27,475 Stage-5 map = 100%,  reduce = 54%, Cumulative CPU 615.06 sec
2016-08-12 19:34:28,552 Stage-5 map = 100%,  reduce = 58%, Cumulative CPU 627.21 sec
2016-08-12 19:34:29,626 Stage-5 map = 100%,  reduce = 60%, Cumulative CPU 636.3 sec
2016-08-12 19:34:30,690 Stage-5 map = 100%,  reduce = 63%, Cumulative CPU 648.52 sec
2016-08-12 19:34:31,757 Stage-5 map = 100%,  reduce = 65%, Cumulative CPU 659.35 sec
2016-08-12 19:34:32,824 Stage-5 map = 100%,  reduce = 66%, Cumulative CPU 673.33 sec
2016-08-12 19:34:33,897 Stage-5 map = 100%,  reduce = 67%, Cumulative CPU 699.63 sec
2016-08-12 19:34:34,965 Stage-5 map = 100%,  reduce = 68%, Cumulative CPU 722.55 sec
2016-08-12 19:34:36,030 Stage-5 map = 100%,  reduce = 71%, Cumulative CPU 754.03 sec
2016-08-12 19:34:37,098 Stage-5 map = 100%,  reduce = 73%, Cumulative CPU 760.98 sec
2016-08-12 19:34:38,164 Stage-5 map = 100%,  reduce = 75%, Cumulative CPU 767.18 sec
2016-08-12 19:34:39,239 Stage-5 map = 100%,  reduce = 80%, Cumulative CPU 789.09 sec
2016-08-12 19:34:40,310 Stage-5 map = 100%,  reduce = 82%, Cumulative CPU 795.42 sec
2016-08-12 19:34:41,379 Stage-5 map = 100%,  reduce = 86%, Cumulative CPU 804.74 sec
2016-08-12 19:34:42,447 Stage-5 map = 100%,  reduce = 90%, Cumulative CPU 819.99 sec
2016-08-12 19:34:43,521 Stage-5 map = 100%,  reduce = 94%, Cumulative CPU 832.86 sec
2016-08-12 19:34:45,664 Stage-5 map = 100%,  reduce = 97%, Cumulative CPU 842.42 sec
2016-08-12 19:34:48,876 Stage-5 map = 100%,  reduce = 99%, Cumulative CPU 848.86 sec
2016-08-12 19:34:49,947 Stage-5 map = 100%,  reduce = 100%, Cumulative CPU 850.51 sec
MapReduce Total cumulative CPU time: 14 minutes 10 seconds 510 msec
Ended Job = job_1470994648796_0014
Stage-23 is filtered out by condition resolver.
Stage-24 is filtered out by condition resolver.
Stage-1 is selected by condition resolver.
Launching Job 7 out of 15
Number of reduce tasks not specified. Estimated from input data size: 2
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 19:34:54 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:34:54 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:34:54 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:36:42 INFO input.FileInputFormat: Total input paths to process : 11
16/08/12 19:36:42 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 0
16/08/12 19:36:42 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
16/08/12 19:36:44 INFO mapreduce.JobSubmitter: number of splits:8
16/08/12 19:36:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0015
16/08/12 19:36:45 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0015
16/08/12 19:36:45 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0015/
Starting Job = job_1470994648796_0015, Tracking URL = http://node643:8088/proxy/application_1470994648796_0015/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0015
Hadoop job information for Stage-1: number of mappers: 8; number of reducers: 2
16/08/12 19:37:16 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:37:16,292 Stage-1 map = 0%,  reduce = 0%
2016-08-12 19:37:35,169 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 14.56 sec
2016-08-12 19:37:41,597 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 53.87 sec
2016-08-12 19:37:43,736 Stage-1 map = 29%,  reduce = 0%, Cumulative CPU 89.11 sec
2016-08-12 19:37:44,804 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 113.89 sec
2016-08-12 19:37:45,879 Stage-1 map = 76%,  reduce = 0%, Cumulative CPU 131.11 sec
2016-08-12 19:37:48,017 Stage-1 map = 85%,  reduce = 0%, Cumulative CPU 143.2 sec
2016-08-12 19:37:49,088 Stage-1 map = 92%,  reduce = 0%, Cumulative CPU 148.04 sec
2016-08-12 19:37:50,165 Stage-1 map = 96%,  reduce = 0%, Cumulative CPU 150.29 sec
2016-08-12 19:37:53,395 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 154.92 sec
2016-08-12 19:38:07,284 Stage-1 map = 100%,  reduce = 21%, Cumulative CPU 160.38 sec
2016-08-12 19:38:10,486 Stage-1 map = 100%,  reduce = 26%, Cumulative CPU 163.47 sec
2016-08-12 19:38:11,551 Stage-1 map = 100%,  reduce = 49%, Cumulative CPU 170.58 sec
2016-08-12 19:38:13,687 Stage-1 map = 100%,  reduce = 58%, Cumulative CPU 176.84 sec
2016-08-12 19:38:15,835 Stage-1 map = 100%,  reduce = 61%, Cumulative CPU 180.33 sec
2016-08-12 19:38:16,910 Stage-1 map = 100%,  reduce = 66%, Cumulative CPU 183.4 sec
2016-08-12 19:38:19,049 Stage-1 map = 100%,  reduce = 68%, Cumulative CPU 191.22 sec
2016-08-12 19:38:20,117 Stage-1 map = 100%,  reduce = 69%, Cumulative CPU 197.44 sec
2016-08-12 19:38:22,261 Stage-1 map = 100%,  reduce = 75%, Cumulative CPU 200.83 sec
2016-08-12 19:38:23,328 Stage-1 map = 100%,  reduce = 79%, Cumulative CPU 207.33 sec
2016-08-12 19:38:25,469 Stage-1 map = 100%,  reduce = 86%, Cumulative CPU 210.45 sec
2016-08-12 19:38:26,535 Stage-1 map = 100%,  reduce = 92%, Cumulative CPU 213.69 sec
2016-08-12 19:38:27,599 Stage-1 map = 100%,  reduce = 94%, Cumulative CPU 215.77 sec
2016-08-12 19:38:29,744 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 219.08 sec
MapReduce Total cumulative CPU time: 3 minutes 39 seconds 80 msec
Ended Job = job_1470994648796_0015
Launching Job 8 out of 15
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 19:38:33 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:38:33 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:38:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:40:18 INFO input.FileInputFormat: Total input paths to process : 2
16/08/12 19:40:18 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 4, size left: 0
16/08/12 19:40:19 INFO mapreduce.JobSubmitter: number of splits:2
16/08/12 19:40:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0016
16/08/12 19:40:20 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0016
16/08/12 19:40:20 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0016/
Starting Job = job_1470994648796_0016, Tracking URL = http://node643:8088/proxy/application_1470994648796_0016/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0016
Hadoop job information for Stage-2: number of mappers: 2; number of reducers: 1
16/08/12 19:40:44 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:40:44,851 Stage-2 map = 0%,  reduce = 0%
2016-08-12 19:40:54,795 Stage-2 map = 50%,  reduce = 0%, Cumulative CPU 3.07 sec
2016-08-12 19:41:07,683 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 6.15 sec
2016-08-12 19:41:23,737 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 10.58 sec
MapReduce Total cumulative CPU time: 10 seconds 580 msec
Ended Job = job_1470994648796_0016
Launching Job 9 out of 15
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 19:41:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:41:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:41:28 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 19:43:12 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 19:43:12 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
16/08/12 19:43:13 INFO mapreduce.JobSubmitter: number of splits:1
16/08/12 19:43:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0017
16/08/12 19:43:14 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0017
16/08/12 19:43:14 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0017/
Starting Job = job_1470994648796_0017, Tracking URL = http://node643:8088/proxy/application_1470994648796_0017/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0017
Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 1
16/08/12 19:43:39 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 19:43:39,872 Stage-3 map = 0%,  reduce = 0%
2016-08-12 19:44:04,031 Stage-3 map = 100%,  reduce = 0%, Cumulative CPU 3.2 sec
2016-08-12 19:44:26,559 Stage-3 map = 100%,  reduce = 100%, Cumulative CPU 8.11 sec
MapReduce Total cumulative CPU time: 8 seconds 110 msec
Ended Job = job_1470994648796_0017
Loading data to table default.q5_local_supplier_volume
16/08/12 19:44:30 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
16/08/12 19:44:30 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
MapReduce Jobs Launched: 
Stage-Stage-22: Map: 1   Cumulative CPU: 6.64 sec   HDFS Read: 7979 HDFS Write: 222 SUCCESS
Stage-Stage-19: Map: 1   Cumulative CPU: 9.47 sec   HDFS Read: 14182336 HDFS Write: 577426 SUCCESS
Stage-Stage-16: Map: 29   Cumulative CPU: 698.01 sec   HDFS Read: 7776171787 HDFS Write: 557334620 SUCCESS
Stage-Stage-5: Map: 15  Reduce: 10   Cumulative CPU: 850.51 sec   HDFS Read: 2306747760 HDFS Write: 83207781 SUCCESS
Stage-Stage-1: Map: 8  Reduce: 2   Cumulative CPU: 219.08 sec   HDFS Read: 328125221 HDFS Write: 514 SUCCESS
Stage-Stage-2: Map: 2  Reduce: 1   Cumulative CPU: 10.58 sec   HDFS Read: 7927 HDFS Write: 257 SUCCESS
Stage-Stage-3: Map: 1  Reduce: 1   Cumulative CPU: 8.11 sec   HDFS Read: 6074 HDFS Write: 223 SUCCESS
Total MapReduce CPU Time Spent: 30 minutes 2 seconds 400 msec
OK
Time taken: 1446.725 seconds

    ——set hive.execution.engine=tez

    备注：在.hive文件里设置了为tez调度模式。

lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q5_local_supplier_volume.hive 
which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 19:09:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
16/08/12 19:09:31 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/08/12 19:09:39 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/08/12 19:09:39 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
Query ID = lwh_20160812190931_c82a04f5-da59-4eb2-a814-4f295ace94f1
Total jobs = 1
Launching Job 1 out of 1
16/08/12 19:09:39 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/08/12 19:11:01 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:11:02 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0010
16/08/12 19:11:28 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080


Status: Running (Executing on YARN cluster with App id application_1470994648796_0010)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED  
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED      1          1        0        0       0       0  
Map 10 ......... container     SUCCEEDED      1          1        0        0       0       0  
Map 11 ......... container     SUCCEEDED      1          1        0        0       0       0  
Map 5 .......... container     SUCCEEDED      4          4        0        0       0       0  
Map 7 .......... container     SUCCEEDED     15         15        0        0       0       0  
Map 9 .......... container     SUCCEEDED      1          1        0        0       0       0  
Reducer 2 ...... container     SUCCEEDED     38         38        0        0       0       0  
Reducer 3 ...... container     SUCCEEDED     41         41        0        0       0       0  
Reducer 4 ...... container     SUCCEEDED      1          1        0        0       0       0  
Reducer 6 ...... container     SUCCEEDED     35         35        0        0       0       0  
Reducer 8 ...... container     SUCCEEDED     31         31        0        0       0       0  
----------------------------------------------------------------------------------------------
VERTICES: 11/11  [==========================>>] 100%  ELAPSED TIME: 218.63 s   
----------------------------------------------------------------------------------------------
Loading data to table default.q5_local_supplier_volume
16/08/12 19:15:08 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.
OK
Time taken: 339.425 seconds
 Hive-on-MRHive-on-Tez
q1436.074s189.653s
q51446.725s339.425s

 

②并行查询q1+q2

    ——hive.execution.engine=tez

    q1:

lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q1_pricing_summary_report.hive 
which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 19:56:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
16/08/12 19:56:08 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/08/12 19:56:17 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
16/08/12 19:56:17 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
Query ID = lwh_20160812195608_c1cde9c4-c451-4230-8515-92f1be8758ee
Total jobs = 1
Launching Job 1 out of 1
16/08/12 19:56:17 INFO Configuration.deprecation: mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
16/08/12 19:58:51 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 19:58:53 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0018
16/08/12 19:59:22 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080


Status: Running (Executing on YARN cluster with App id application_1470994648796_0018)

----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED  
----------------------------------------------------------------------------------------------
Map 1 .......... container     SUCCEEDED     15         15        0        0       0       0  
Reducer 2 ...... container     SUCCEEDED     11         11        0        0       0       0  
Reducer 3 ...... container     SUCCEEDED      1          1        0        0       0       0  
----------------------------------------------------------------------------------------------
VERTICES: 03/03  [==========================>>] 100%  ELAPSED TIME: 136.81 s   
----------------------------------------------------------------------------------------------
OK
16/08/12 20:01:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 20:01:41 INFO mapred.FileInputFormat: Total input paths to process : 1
A       F       3.77518399E8    5.660657277972496E11    5.377591042780614E11    5.592766708921191E11    25.500975103007097      38237.15100895852       0.05000657454011058     14804077
N       F       9851614.0       1.4767438399170017E10   1.4028805792211395E10   1.459049099836676E10    25.522448302840946      38257.81066008119       0.04997336773764426     385998
N       O       7.43124873E8    1.1143022869018733E12   1.0585809221449624E12   1.1009370001705708E12   25.498075870689316      38233.90292348158       0.050000811821093484    29144351
R       F       3.7773283E8     5.664310549759983E11    5.381109226647607E11    5.596347808850894E11    25.50838478968014       38251.21927355965       0.04999679231395468     14808183
Time taken: 332.832 seconds, Fetched: 4 row(s)

    q2:

输出信息过多，没有收集到。

    ——hive.execution.engine=tez

    q1:

lwh@node643:~/hive> bin/hive -f ~/TPC-H_on_Hive/tpch_lyx/q1_pricing_summary_report.hive 
which: no hbase in (/home/lwh/hive/bin:/home/lwh/jdk/bin:/home/lwh/hadoop-2.7.2/bin:/home/lwh/jdk/bin:/home/lwh/asterixdb/bin:/dawnfs/software/mpi/openmpi1.4.3-intel/bin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/bin/intel64:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/home/lwh/jdk1.7.0_17/bin:/home/lwh/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/dawnfs/software/intel/composerxe-2011.3.174/composerxe-2011.3.174/mpirt/bin/intel64)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hive/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/lwh/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
16/08/12 20:16:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Logging initialized using configuration in file:/home/lwh/hive/conf/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
16/08/12 20:16:27 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
16/08/12 20:16:32 ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = lwh_20160812201627_2ccd2f24-ad79-466b-8ad9-ff0f7b7cd191
Total jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 31
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 20:16:37 INFO Configuration.deprecation: mapred.submit.replication is deprecated. Instead, use mapreduce.client.submit.file.replication
16/08/12 20:16:38 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 20:16:39 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 20:16:40 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 20:19:45 INFO input.FileInputFormat: Total input paths to process : 1
16/08/12 20:19:45 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 268435456
16/08/12 20:19:47 INFO mapreduce.JobSubmitter: number of splits:29
16/08/12 20:19:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0022
16/08/12 20:19:49 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0022
16/08/12 20:19:49 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0022/
Starting Job = job_1470994648796_0022, Tracking URL = http://node643:8088/proxy/application_1470994648796_0022/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0022
Hadoop job information for Stage-1: number of mappers: 29; number of reducers: 31
16/08/12 20:20:16 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 20:20:16,607 Stage-1 map = 0%,  reduce = 0%
2016-08-12 20:20:40,816 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 34.83 sec
2016-08-12 20:20:47,294 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 170.63 sec
2016-08-12 20:20:53,777 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 290.64 sec
2016-08-12 20:20:55,943 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 313.98 sec
2016-08-12 20:20:57,024 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU 335.71 sec
2016-08-12 20:21:00,260 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 395.76 sec
2016-08-12 20:21:01,334 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 407.72 sec
2016-08-12 20:21:02,413 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 425.55 sec
2016-08-12 20:21:03,498 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU 433.11 sec
2016-08-12 20:21:08,897 Stage-1 map = 43%,  reduce = 0%, Cumulative CPU 456.3 sec
2016-08-12 20:21:09,970 Stage-1 map = 45%,  reduce = 0%, Cumulative CPU 462.23 sec
2016-08-12 20:21:12,115 Stage-1 map = 47%,  reduce = 0%, Cumulative CPU 467.86 sec
2016-08-12 20:21:13,187 Stage-1 map = 49%,  reduce = 0%, Cumulative CPU 482.36 sec
2016-08-12 20:21:15,356 Stage-1 map = 49%,  reduce = 1%, Cumulative CPU 522.83 sec
2016-08-12 20:21:16,430 Stage-1 map = 52%,  reduce = 1%, Cumulative CPU 528.38 sec
2016-08-12 20:21:21,816 Stage-1 map = 53%,  reduce = 1%, Cumulative CPU 603.66 sec
2016-08-12 20:21:22,889 Stage-1 map = 54%,  reduce = 1%, Cumulative CPU 616.1 sec
2016-08-12 20:21:23,962 Stage-1 map = 56%,  reduce = 1%, Cumulative CPU 626.38 sec
2016-08-12 20:21:27,195 Stage-1 map = 57%,  reduce = 1%, Cumulative CPU 665.39 sec
2016-08-12 20:21:28,270 Stage-1 map = 59%,  reduce = 1%, Cumulative CPU 677.06 sec
2016-08-12 20:21:29,346 Stage-1 map = 60%,  reduce = 2%, Cumulative CPU 707.06 sec
2016-08-12 20:21:31,495 Stage-1 map = 67%,  reduce = 2%, Cumulative CPU 733.66 sec
2016-08-12 20:21:33,657 Stage-1 map = 72%,  reduce = 2%, Cumulative CPU 753.21 sec
2016-08-12 20:21:36,878 Stage-1 map = 77%,  reduce = 2%, Cumulative CPU 773.63 sec
2016-08-12 20:21:37,945 Stage-1 map = 78%,  reduce = 2%, Cumulative CPU 777.17 sec
2016-08-12 20:21:39,026 Stage-1 map = 79%,  reduce = 2%, Cumulative CPU 780.76 sec
2016-08-12 20:21:40,103 Stage-1 map = 82%,  reduce = 2%, Cumulative CPU 783.92 sec
2016-08-12 20:21:42,241 Stage-1 map = 82%,  reduce = 3%, Cumulative CPU 790.95 sec
2016-08-12 20:21:44,376 Stage-1 map = 84%,  reduce = 3%, Cumulative CPU 795.82 sec
2016-08-12 20:21:46,535 Stage-1 map = 86%,  reduce = 4%, Cumulative CPU 835.79 sec
2016-08-12 20:21:48,681 Stage-1 map = 86%,  reduce = 5%, Cumulative CPU 851.77 sec
2016-08-12 20:21:49,756 Stage-1 map = 86%,  reduce = 6%, Cumulative CPU 861.69 sec
2016-08-12 20:21:55,102 Stage-1 map = 87%,  reduce = 6%, Cumulative CPU 882.42 sec
2016-08-12 20:21:56,174 Stage-1 map = 89%,  reduce = 8%, Cumulative CPU 888.73 sec
2016-08-12 20:21:57,245 Stage-1 map = 89%,  reduce = 9%, Cumulative CPU 890.14 sec
2016-08-12 20:22:00,466 Stage-1 map = 89%,  reduce = 10%, Cumulative CPU 903.81 sec
2016-08-12 20:22:02,603 Stage-1 map = 91%,  reduce = 10%, Cumulative CPU 914.72 sec
2016-08-12 20:22:04,740 Stage-1 map = 93%,  reduce = 11%, Cumulative CPU 920.01 sec
2016-08-12 20:22:07,945 Stage-1 map = 94%,  reduce = 11%, Cumulative CPU 924.0 sec
2016-08-12 20:22:10,087 Stage-1 map = 95%,  reduce = 11%, Cumulative CPU 925.81 sec
2016-08-12 20:22:14,371 Stage-1 map = 95%,  reduce = 12%, Cumulative CPU 932.02 sec
2016-08-12 20:22:16,506 Stage-1 map = 95%,  reduce = 13%, Cumulative CPU 936.41 sec
2016-08-12 20:22:28,287 Stage-1 map = 98%,  reduce = 13%, Cumulative CPU 950.75 sec
2016-08-12 20:22:29,355 Stage-1 map = 100%,  reduce = 13%, Cumulative CPU 952.36 sec
2016-08-12 20:22:30,424 Stage-1 map = 100%,  reduce = 18%, Cumulative CPU 954.69 sec
2016-08-12 20:22:31,492 Stage-1 map = 100%,  reduce = 23%, Cumulative CPU 963.74 sec
2016-08-12 20:22:32,562 Stage-1 map = 100%,  reduce = 30%, Cumulative CPU 980.79 sec
2016-08-12 20:22:33,632 Stage-1 map = 100%,  reduce = 41%, Cumulative CPU 1017.66 sec
2016-08-12 20:22:34,704 Stage-1 map = 100%,  reduce = 42%, Cumulative CPU 1023.84 sec
2016-08-12 20:22:38,973 Stage-1 map = 100%,  reduce = 45%, Cumulative CPU 1031.16 sec
2016-08-12 20:22:40,041 Stage-1 map = 100%,  reduce = 48%, Cumulative CPU 1036.93 sec
2016-08-12 20:22:41,118 Stage-1 map = 100%,  reduce = 52%, Cumulative CPU 1042.65 sec
2016-08-12 20:22:43,278 Stage-1 map = 100%,  reduce = 55%, Cumulative CPU 1049.32 sec
2016-08-12 20:22:44,355 Stage-1 map = 100%,  reduce = 68%, Cumulative CPU 1077.92 sec
2016-08-12 20:22:45,432 Stage-1 map = 100%,  reduce = 81%, Cumulative CPU 1099.9 sec
2016-08-12 20:22:46,501 Stage-1 map = 100%,  reduce = 84%, Cumulative CPU 1105.43 sec
2016-08-12 20:22:49,707 Stage-1 map = 100%,  reduce = 87%, Cumulative CPU 1111.84 sec
2016-08-12 20:22:50,778 Stage-1 map = 100%,  reduce = 94%, Cumulative CPU 1124.73 sec
2016-08-12 20:22:51,847 Stage-1 map = 100%,  reduce = 97%, Cumulative CPU 1131.81 sec
2016-08-12 20:22:55,050 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 1139.38 sec
MapReduce Total cumulative CPU time: 18 minutes 59 seconds 380 msec
Ended Job = job_1470994648796_0022
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
16/08/12 20:23:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 20:23:03 INFO client.RMProxy: Connecting to ResourceManager at node643/10.10.20.43:8080
16/08/12 20:23:03 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
16/08/12 20:25:38 INFO input.FileInputFormat: Total input paths to process : 31
16/08/12 20:25:38 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 8, size left: 0
16/08/12 20:25:40 INFO mapreduce.JobSubmitter: number of splits:6
16/08/12 20:25:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1470994648796_0025
16/08/12 20:25:41 INFO impl.YarnClientImpl: Submitted application application_1470994648796_0025
16/08/12 20:25:41 INFO mapreduce.Job: The url to track the job: http://node643:8088/proxy/application_1470994648796_0025/
Starting Job = job_1470994648796_0025, Tracking URL = http://node643:8088/proxy/application_1470994648796_0025/
Kill Command = /home/lwh/hadoop-2.7.2/bin/hadoop job  -kill job_1470994648796_0025
Hadoop job information for Stage-2: number of mappers: 6; number of reducers: 1
16/08/12 20:26:06 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2016-08-12 20:26:06,419 Stage-2 map = 0%,  reduce = 0%
2016-08-12 20:26:16,449 Stage-2 map = 17%,  reduce = 0%, Cumulative CPU 3.61 sec
2016-08-12 20:26:28,243 Stage-2 map = 33%,  reduce = 0%, Cumulative CPU 7.08 sec
2016-08-12 20:26:29,314 Stage-2 map = 67%,  reduce = 0%, Cumulative CPU 14.63 sec
2016-08-12 20:26:32,526 Stage-2 map = 100%,  reduce = 0%, Cumulative CPU 21.77 sec
2016-08-12 20:26:35,737 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 27.16 sec
MapReduce Total cumulative CPU time: 27 seconds 160 msec
Ended Job = job_1470994648796_0025
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 29  Reduce: 31   Cumulative CPU: 1139.38 sec   HDFS Read: 7776522714 HDFS Write: 3305 SUCCESS
Stage-Stage-2: Map: 6  Reduce: 1   Cumulative CPU: 27.16 sec   HDFS Read: 39220 HDFS Write: 723 SUCCESS
Total MapReduce CPU Time Spent: 19 minutes 26 seconds 540 msec
OK
16/08/12 20:26:41 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
16/08/12 20:26:41 INFO mapred.FileInputFormat: Total input paths to process : 1
A       F       3.77518399E8    5.660657277972468E11    5.3775910427806824E11   5.592766708921185E11    25.500975103007097      38237.15100895833       0.05000657454026939     14804077
N       F       9851614.0       1.4767438399169981E10   1.4028805792211395E10   1.4590490998366728E10   25.522448302840946      38257.81066008109       0.04997336773765435     385998
N       O       7.43124873E8    1.1143022869018765E12   1.0585809221449629E12   1.100937000170583E12    25.498075870689316      38233.90292348169       0.050000811820997394    29144351
R       F       3.7773283E8     5.664310549759971E11    5.381109226647635E11    5.596347808850863E11    25.50838478968014       38251.21927355956       0.04999679231411408     14808183
Time taken: 614.032 seconds, Fetched: 4 row(s)

    q2:

Time taken: 968.034 seconds, Fetched: 6351 row(s)
 Hive-on-MRHive-on-Tez
q1614.032s332.832s
q2968.034s/
total968.034s332.832s

 

转载于:https://my.oschina.net/whulyx/blog/731940

ci1652315
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
hive中通用调优-fetch抓取-mr本地模式
mx13156的博客
09-01 562
mr本地模式：如果非要执行mr，能够执行本地的就不要提交yarn执行，默认是关闭的，意味着只要走mr就提交到yarn。Fetch抓取机制：在执行SQL的时候，能不走mr就不走mr，尽量直接去操作数据文件。hive提供了一个参数，自动切换mr程序为本地模式，如果不满条件，就执行yarn模式。参数设置：hive.fetch.task.conversion= more。mapreduce.framework.name = local 本地模式。通常情况下，全局查找，字段查找，limit查找都不走mr程序。
Hive基于MR/Tez/本地模式的对比
九筒的博客
04-15 1201
实验环境 内存：16GB CPU：i5 4590 Centos：6.8 Hive：2.3.6 实验数据 数据量：1138526 字段：3个 实验目的 对比Hive on MR 与 Hive on Tez 以及Hive本地模式在不同语句下的执行效率对比 实验结论 省去看实验过程的麻烦 本地模式肉眼可见，效果极佳，但是对硬件和内存有一定要求，无法处理大数据量 Tez优势是将具有依赖的多个作业转换成一个作业，从而减少与HDFS的交互，达到提升效率的目的。 count和排序优化效果不好的原因是什么——尚未找到合
Hive on Spark VS Spark on Hive
u012667450的博客
12-10 5278
Hive on Spark VS Spark on Hive 两者概述 Hive on Spark Hive on Spark是由Cloudera发起，由Intel、MapR等公司共同参与的开源项目，其目的是把Spark作为Hive的一个计算引擎，将Hive的查询作为Spark的任务提交到Spark集群上进行计算。通过该项目，可以提高Hive查询的性能，同时为已经部署了Hive或者Spark的用户提供了更加灵活的选择，从而进一步提高Hive和Spark的普及率。 Hive on spark大体与SparkS
Hive 使用TEZ引擎导致连接Hive SHELL慢的问题
数据源的港湾
07-06 3930
在使用HDP Hadoop版本时，Ambari界面允许选择Hive执行引擎是MapReduce还是TEZ，如下图所示 使用TEZ作为Hive执行引擎来替代MapReduce，可以在很多场景上实现更好的效率提高，然后使用TEZ作为默认引擎也会导致一些问题。最近就碰到这样的场景，客户使用TEZ作为执行引擎，在计算过程中确实比使用默认的MapReduce快很多，但遇到的一个问题是：每次总要花10~...
hive on spark 以及 hive on mr
weixin_42307036的博客
11-18 2870
--默认方式是mapreduce set hive.execution.engine=spark; select count(*) from sxs_comment_his; set hive.execution.engine=MR; select count(*) from sxs_comment_his; 总结： hive默认方式是mapreduce，比on spark 慢 从最后一张图可以看到一个箭头所覆盖的ip，可以通过这个ip访问YARN的UI界面，可能需要个跳板机，比如我司还需要登陆个wi
docker-hive-on-tez:在 Tez 上运行的 Apache Hive 的 Docker 镜像
06-10
在 Tez 上运行 Apache Hive 的 Docker 镜像此存储库包含一个 docker 文件，用于构建 docker 映像以在 Tez 上运行 Apache Hive。 这个 docker 文件依赖于我的其他包含和 基础镜像的存储库。当前版本Apache Hive（主干...
hive-jdbc-1.2.1-standalone.jar
10-21
dbeaver-cli.exe客户端连接hive所需要的其中一个jar包 hive-jdbc-1.2.1-standalone.jar
hive-jdbc-3.1.1-standalone.jar
06-21
hive版本匹配的驱动，该驱动可以在hive的jdbc目录下找到
hive-hbase-handler-1.2.1.jar
09-04
被编译的hive-hbase-handler-1.2.1.jar，用于在Hive中创建关联HBase表的jar，解决创建Hive关联HBase时报FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.apache.hadoop....
apache-hive-2.1.0-bin.tar.zip
05-04
apache-hive-2.1.0-bin
大数据Hive on MR/TEZ与hadoop的整合应用
01-15
本课程主要介绍hive的架构与应用，首先介绍了hive的实现结构，然后介绍了Hive Metastore三种模式，接着重点介绍hive的安装与配置，以及hive sql的基本操作，后面又介绍了hive beeline、hive on tez的使用，并测试了hive on MR和TEZ的性能优缺。本课程以apache hadoop发行版本为例，介绍如何实现hadoop与hive的整合，hadoop环境采用手动安装的hadoop3.2.1环境，hive版本同样采用apache hive3.1.2发行版本。以3台主机为例作为hive安装环境，采用Centos7.7操作系统。课程大纲如下：    1、hive的架构与应用场景2、Hive Metastore三种运行模式3、hive安装以及与hadoop整合4、hive的安装与配置5、启动metastore与hiveserver2服务  6、hive常用SQL操作7、beeline的使用8、tez的编译以及tez与hive整合步骤9、测试tez功能以及与MR性能比较
解决：WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions.
热门推荐
lanachen的博客
07-07 1万+
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive on mr参数配置
weiha666的博客
12-23 2266
MR参数： set mapreduce.job.queuename=queue_name; #设置作业提交队列 set hive.execution.engine=mr; #设置计算引擎 set mapreduce.map.memory.mb=4096; #设置map内存 set mapreduce...
beeline执行sql语句_jdbc 连hive查询报错，但beeline/hive cli都能正确执行
weixin_29977617的博客
12-23 1209
使用的各版本如下：hive-0.13.1-cdh5.3.2hadoop-2.5.0-cdh5.3.2hive开启服务器:nohup hive --servicehiveserver2 &java应用通过jdbc方式连接hive, url:jdbc:hive2://10.199.201.41:10000/default当执行 select * from kylin_account where...
Hive_引擎MR、Tez、Spark
qq_56870570的博客
08-20 5684
1.Hive引擎简介 Hive引擎包括：默认MR、Tez、Spark 不更换引擎hive默认的是MR。 Hive on Spark：Hive既作为存储元数据又负责SQL的解析优化，语法是HQL语法，执行引擎变成了Spark，Spark负责采用RDD执行。 Spark on Hive : Hive只作为存储元数据，Spark负责SQL解析优化，语法是SparkSQL语法，Spark负责采用RDD执行。 2. Spark引擎 2.1HiveonSpark配置 1）兼容性说明 注意：...
hive on tez hive运行在tez之上 安装测试
duguyiren3476的专栏
05-26 1249
hive on tez详细配置和运行测试 tez hadoop hive hdfs yarn 环境: hadoop-2.5.2 hive-0.14 tez-0.5.3 hive on tez 的方式有两种安装配置方式: 在hadoop中配置 在hive中配置 比较: 第二种方式:当已经有了稳定的hadoop集群,而不想动这个集群时,可以考虑采用第二种方式配置...
HIVE学习二：hive on tez
patrick_big_data的博客
03-02 909
文章目录环境介绍下载、安装、配置TEZ测试hive on tez参考网址 环境介绍 组件 版本 hadoop 2.6.5 hive 2.3.6 tez 0.8.5 tez对hadoop版本是有要求的。tez 0.8及以上需要hadoop 2.6及以上。tez 0.9及以上需要hadoop 2.7及以上。 下载、安装、配置TEZ 从清华镜像站下载对应版本的tez如apa...
Hive on Tez
一名假的程序猿
08-13 2737
参考 http://tez.apache.org/install.html https://cwiki.apache.org/confluence/display/Hive/Hive+on+Tez https://cwiki.apache.org/confluence/display/Hive/Hive-Tez+Compatibility https://github.com/apache...
hive-site。xml配置文件
最新发布
07-08
- `hive.execution.engine`: Hive执行引擎（如MR、Tez）。 - `hive.tez.container.size`: Tez容器的内存大小。 这些只是一小部分可能出现在`hive-site.xml`中的配置项，具体取决于你的Hive版本和需求。你可以根据...

“相关推荐”对你有帮助么？

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值