Hive 执行日志

深度学习V666

已于 2024-06-28 11:27:06 修改

阅读量137

点赞数 2

分类专栏： Hadoop 文章标签： hive hadoop 数据仓库

于 2024-06-28 08:51:04 首次发布

本文链接：https://blog.csdn.net/pansiliang2009/article/details/140032581

版权

Hadoop 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

INFO  : 2024-06-27 19:48:29,508 Stage-2 map = 100%,  reduce = 100%, Cumulative CPU 21682.65 sec
INFO  : MapReduce Total cumulative CPU time: 0 days 6 hours 1 minutes 22 seconds 650 msec
INFO  : Ended Job = job_1717754847593_24584
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-1: Map: 525  Reduce: 1099   Cumulative CPU: 14385.69 sec   HDFS Read: 140827899324 HDFS Write: 16236173571 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-3: Map: 173  Reduce: 776   Cumulative CPU: 12376.94 sec   HDFS Read: 22315158039 HDFS Write: 91413883474 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-11: Map: 79  Reduce: 321   Cumulative CPU: 10433.71 sec   HDFS Read: 7887307487 HDFS Write: 16941450267 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-12: Map: 13  Reduce: 54   Cumulative CPU: 470.17 sec   HDFS Read: 276127341 HDFS Write: 1243186004 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-13: Map: 5  Reduce: 12   Cumulative CPU: 213.71 sec   HDFS Read: 527260064 HDFS Write: 1502131007 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-14: Map: 4  Reduce: 1   Cumulative CPU: 64.06 sec   HDFS Read: 36071434 HDFS Write: 11899330 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-16: Map: 1  Reduce: 1   Cumulative CPU: 12.56 sec   HDFS Read: 19359533 HDFS Write: 12528312 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-18: Map: 4  Reduce: 2   Cumulative CPU: 52.18 sec   HDFS Read: 16423138 HDFS Write: 120166293 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-15: Map: 1  Reduce: 1   Cumulative CPU: 13.87 sec   HDFS Read: 11905662 HDFS Write: 12305073 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-17: Map: 1  Reduce: 1   Cumulative CPU: 12.33 sec   HDFS Read: 12534653 HDFS Write: 12911102 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-4: Map: 71  Reduce: 1099   Cumulative CPU: 9181.86 sec   HDFS Read: 7893855061 HDFS Write: 2942184967 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-8: Map: 71  Reduce: 1099   Cumulative CPU: 8780.35 sec   HDFS Read: 7636431245 HDFS Write: 552806637 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-19: Map: 589  Reduce: 1099   Cumulative CPU: 19345.59 sec   HDFS Read: 147817148501 HDFS Write: 24867610123 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-5: Map: 12  Reduce: 44   Cumulative CPU: 445.32 sec   HDFS Read: 2942681919 HDFS Write: 2592041620 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-9: Map: 5  Reduce: 9   Cumulative CPU: 151.99 sec   HDFS Read: 553136971 HDFS Write: 464639335 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-20: Map: 92  Reduce: 371   Cumulative CPU: 4501.78 sec   HDFS Read: 24870101598 HDFS Write: 20971070579 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-6: Map: 13  Reduce: 46   Cumulative CPU: 448.62 sec   HDFS Read: 3056886250 HDFS Write: 2992629790 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-21: Map: 139  Reduce: 1099   Cumulative CPU: 10973.64 sec   HDFS Read: 34826082499 HDFS Write: 31343751931 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-2: Map: 531  Reduce: 1099   Cumulative CPU: 21682.65 sec   HDFS Read: 161852390435 HDFS Write: 0 HDFS EC Read: 0 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 1 days 7 hours 32 minutes 27 seconds 20 msec
INFO  : Completed executing command(queryId=hive_20240627165913_d652b9da-2fc5-4f91-b896-123ddf6f7575); Time taken: 10133.492 seconds
INFO  : OK
79,853,984 rows affected (10157.077 seconds)

demo2

INFO  : 2024-06-28 09:20:17,705 Stage-6 map = 100%,  reduce = 0%, Cumulative CPU 279.07 sec
INFO  : MapReduce Total cumulative CPU time: 4 minutes 39 seconds 70 msec
INFO  : Ended Job = job_1719535099476_0010
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table db_test.db_intellectual_propertyt_patents_new from hdfs://dh01:8020/user/hive/warehouse/db_test.db/db_intellectual_propertyt_patents_new/.hive-staging_hive_2024-06-28_09-07-32_181_2037352161628219065-6/-ext-10000
INFO  : Starting task [Stage-5:STATS] in serial mode
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-12: Map: 126  Reduce: 126   Cumulative CPU: 2772.71 sec   HDFS Read: 33789936142 HDFS Write: 2513928729 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-13: Map: 126  Reduce: 126   Cumulative CPU: 3480.93 sec   HDFS Read: 33789925684 HDFS Write: 17126801917 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-14: Map: 64  Reduce: 64   Cumulative CPU: 2049.69 sec   HDFS Read: 17138053979 HDFS Write: 14924467225 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-1: Map: 113  Reduce: 347   Cumulative CPU: 5463.64 sec   HDFS Read: 24555606340 HDFS Write: 7970701683 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-2: Map: 30  Reduce: 30   Cumulative CPU: 884.4 sec   HDFS Read: 7972073493 HDFS Write: 7963389859 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-3: Map: 30  Reduce: 30   Cumulative CPU: 979.29 sec   HDFS Read: 7966780168 HDFS Write: 7616276579 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-4: Map: 321  Reduce: 331   Cumulative CPU: 11043.91 sec   HDFS Read: 88799951086 HDFS Write: 21201110390 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-6: Map: 74   Cumulative CPU: 279.07 sec   HDFS Read: 21214611758 HDFS Write: 21757591343 HDFS EC Read: 0 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 0 days 7 hours 29 minutes 13 seconds 640 msec
INFO  : Completed executing command(queryId=hive_20240628090732_74fab356-d293-4405-a57c-d45e2320a631); Time taken: 766.694 seconds
INFO  : OK
47,987,245 rows affected (768.182 seconds)

并行优化demo

INFO  : Query ID = hive_20240628111152_79f7458b-5497-4b76-803a-560abca802e5
INFO  : Total jobs = 15
INFO  : Launching Job 1 out of 15
INFO  : Starting task [Stage-1:MAPRED] in parallel
INFO  : Launching Job 2 out of 15
INFO  : Starting task [Stage-15:MAPRED] in parallel
INFO  : Launching Job 3 out of 15
INFO  : Starting task [Stage-16:MAPRED] in parallel
INFO  : Starting task [Stage-27:CONDITIONAL] in parallel
INFO  : Starting task [Stage-32:MAPREDLOCAL] in serial mode
INFO  : Execution completed successfully
INFO  : MapredLocal task succeeded
INFO  : Starting task [Stage-25:CONDITIONAL] in parallel
INFO  : Launching Job 5 out of 15
INFO  : Starting task [Stage-26:MAPRED] in parallel
INFO  : Launching Job 6 out of 15
INFO  : Starting task [Stage-2:MAPRED] in parallel
INFO  : Starting task [Stage-22:CONDITIONAL] in parallel
INFO  : Launching Job 7 out of 15
INFO  : Starting task [Stage-3:MAPRED] in parallel
INFO  : Launching Job 8 out of 15
INFO  : Starting task [Stage-4:MAPRED] in parallel
INFO  : Launching Job 9 out of 15
INFO  : Starting task [Stage-5:MAPRED] in parallel
INFO  : Starting task [Stage-20:CONDITIONAL] in parallel
INFO  : Launching Job 10 out of 15
INFO  : Starting task [Stage-6:MAPRED] in parallel
INFO  : Launching Job 11 out of 15
INFO  : Starting task [Stage-7:MAPRED] in parallel
INFO  : Starting task [Stage-13:CONDITIONAL] in serial mode
INFO  : Stage-10 is selected by condition resolver.
INFO  : Stage-9 is filtered out by condition resolver.
INFO  : Stage-11 is filtered out by condition resolver.
INFO  : Starting task [Stage-10:MOVE] in serial mode
INFO  : Moving data to directory hdfs://dh01:8020/user/hive/warehouse/tdxdb.db/t_group_ent_info/.hive-staging_hive_2024-06-28_11-11-52_726_3253104069358746967-46/-ext-10000 from hdfs://dh01:8020/user/hive/warehouse/tdxdb.db/t_group_ent_info/.hive-staging_hive_2024-06-28_11-11-52_726_3253104069358746967-46/-ext-10008
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table tdxdb.t_group_ent_info from hdfs://dh01:8020/user/hive/warehouse/tdxdb.db/t_group_ent_info/.hive-staging_hive_2024-06-28_11-11-52_726_3253104069358746967-46/-ext-10000
INFO  : Starting task [Stage-8:STATS] in serial mode
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-16: Map: 2  Reduce: 1   Cumulative CPU: 21.78 sec   HDFS Read: 30826969 HDFS Write: 1230069 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-15: Map: 9  Reduce: 9   Cumulative CPU: 145.46 sec   HDFS Read: 377189780 HDFS Write: 218441824 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-1: Map: 9  Reduce: 9   Cumulative CPU: 145.27 sec   HDFS Read: 446904283 HDFS Write: 411352262 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-26: Map: 64   Cumulative CPU: 3235.28 sec   HDFS Read: 14113867094 HDFS Write: 30291640119 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-2: Map: 68  Reduce: 432   Cumulative CPU: 3882.94 sec   HDFS Read: 7760741455 HDFS Write: 94094193 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-3: Map: 8  Reduce: 2   Cumulative CPU: 92.13 sec   HDFS Read: 312683077 HDFS Write: 11310877 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-4: Map: 2  Reduce: 1   Cumulative CPU: 18.65 sec   HDFS Read: 11320365 HDFS Write: 11310781 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-5: Map: 1  Reduce: 1   Cumulative CPU: 10.96 sec   HDFS Read: 11317422 HDFS Write: 1708107 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-6: Map: 105  Reduce: 113   Cumulative CPU: 3787.39 sec   HDFS Read: 30301077693 HDFS Write: 3614603 HDFS EC Read: 0 SUCCESS
INFO  : Stage-Stage-7: Map: 4  Reduce: 1   Cumulative CPU: 29.38 sec   HDFS Read: 3664294 HDFS Write: 1505025 HDFS EC Read: 0 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 0 days 3 hours 9 minutes 29 seconds 240 msec
INFO  : Completed executing command(queryId=hive_20240628111152_79f7458b-5497-4b76-803a-560abca802e5); Time taken: 795.243 seconds
INFO  : OK
17,472 rows affected (796.499 seconds)