INFO : 2024-06-27 19:48:29,508 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 21682.65 sec
INFO : MapReduce Total cumulative CPU time: 0 days 6 hours 1 minutes 22 seconds 650 msec
INFO : Ended Job = job_1717754847593_24584
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-1: Map: 525 Reduce: 1099 Cumulative CPU: 14385.69 sec HDFS Read: 140827899324 HDFS Write: 16236173571 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-3: Map: 173 Reduce: 776 Cumulative CPU: 12376.94 sec HDFS Read: 22315158039 HDFS Write: 91413883474 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-11: Map: 79 Reduce: 321 Cumulative CPU: 10433.71 sec HDFS Read: 7887307487 HDFS Write: 16941450267 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-12: Map: 13 Reduce: 54 Cumulative CPU: 470.17 sec HDFS Read: 276127341 HDFS Write: 1243186004 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-13: Map: 5 Reduce: 12 Cumulative CPU: 213.71 sec HDFS Read: 527260064 HDFS Write: 1502131007 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-14: Map: 4 Reduce: 1 Cumulative CPU: 64.06 sec HDFS Read: 36071434 HDFS Write: 11899330 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-16: Map: 1 Reduce: 1 Cumulative CPU: 12.56 sec HDFS Read: 19359533 HDFS Write: 12528312 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-18: Map: 4 Reduce: 2 Cumulative CPU: 52.18 sec HDFS Read: 16423138 HDFS Write: 120166293 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-15: Map: 1 Reduce: 1 Cumulative CPU: 13.87 sec HDFS Read: 11905662 HDFS Write: 12305073 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-17: Map: 1 Reduce: 1 Cumulative CPU: 12.33 sec HDFS Read: 12534653 HDFS Write: 12911102 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-4: Map: 71 Reduce: 1099 Cumulative CPU: 9181.86 sec HDFS Read: 7893855061 HDFS Write: 2942184967 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-8: Map: 71 Reduce: 1099 Cumulative CPU: 8780.35 sec HDFS Read: 7636431245 HDFS Write: 552806637 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-19: Map: 589 Reduce: 1099 Cumulative CPU: 19345.59 sec HDFS Read: 147817148501 HDFS Write: 24867610123 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-5: Map: 12 Reduce: 44 Cumulative CPU: 445.32 sec HDFS Read: 2942681919 HDFS Write: 2592041620 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-9: Map: 5 Reduce: 9 Cumulative CPU: 151.99 sec HDFS Read: 553136971 HDFS Write: 464639335 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-20: Map: 92 Reduce: 371 Cumulative CPU: 4501.78 sec HDFS Read: 24870101598 HDFS Write: 20971070579 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-6: Map: 13 Reduce: 46 Cumulative CPU: 448.62 sec HDFS Read: 3056886250 HDFS Write: 2992629790 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-21: Map: 139 Reduce: 1099 Cumulative CPU: 10973.64 sec HDFS Read: 34826082499 HDFS Write: 31343751931 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-2: Map: 531 Reduce: 1099 Cumulative CPU: 21682.65 sec HDFS Read: 161852390435 HDFS Write: 0 HDFS EC Read: 0 SUCCESS
INFO : Total MapReduce CPU Time Spent: 1 days 7 hours 32 minutes 27 seconds 20 msec
INFO : Completed executing command(queryId=hive_20240627165913_d652b9da-2fc5-4f91-b896-123ddf6f7575); Time taken: 10133.492 seconds
INFO : OK
79,853,984 rows affected (10157.077 seconds)
demo2
INFO : 2024-06-28 09:20:17,705 Stage-6 map = 100%, reduce = 0%, Cumulative CPU 279.07 sec
INFO : MapReduce Total cumulative CPU time: 4 minutes 39 seconds 70 msec
INFO : Ended Job = job_1719535099476_0010
INFO : Starting task [Stage-0:MOVE] in serial mode
INFO : Loading data to table db_test.db_intellectual_propertyt_patents_new from hdfs://dh01:8020/user/hive/warehouse/db_test.db/db_intellectual_propertyt_patents_new/.hive-staging_hive_2024-06-28_09-07-32_181_2037352161628219065-6/-ext-10000
INFO : Starting task [Stage-5:STATS] in serial mode
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-12: Map: 126 Reduce: 126 Cumulative CPU: 2772.71 sec HDFS Read: 33789936142 HDFS Write: 2513928729 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-13: Map: 126 Reduce: 126 Cumulative CPU: 3480.93 sec HDFS Read: 33789925684 HDFS Write: 17126801917 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-14: Map: 64 Reduce: 64 Cumulative CPU: 2049.69 sec HDFS Read: 17138053979 HDFS Write: 14924467225 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-1: Map: 113 Reduce: 347 Cumulative CPU: 5463.64 sec HDFS Read: 24555606340 HDFS Write: 7970701683 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-2: Map: 30 Reduce: 30 Cumulative CPU: 884.4 sec HDFS Read: 7972073493 HDFS Write: 7963389859 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-3: Map: 30 Reduce: 30 Cumulative CPU: 979.29 sec HDFS Read: 7966780168 HDFS Write: 7616276579 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-4: Map: 321 Reduce: 331 Cumulative CPU: 11043.91 sec HDFS Read: 88799951086 HDFS Write: 21201110390 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-6: Map: 74 Cumulative CPU: 279.07 sec HDFS Read: 21214611758 HDFS Write: 21757591343 HDFS EC Read: 0 SUCCESS
INFO : Total MapReduce CPU Time Spent: 0 days 7 hours 29 minutes 13 seconds 640 msec
INFO : Completed executing command(queryId=hive_20240628090732_74fab356-d293-4405-a57c-d45e2320a631); Time taken: 766.694 seconds
INFO : OK
47,987,245 rows affected (768.182 seconds)
并行优化demo
INFO : Query ID = hive_20240628111152_79f7458b-5497-4b76-803a-560abca802e5
INFO : Total jobs = 15
INFO : Launching Job 1 out of 15
INFO : Starting task [Stage-1:MAPRED] in parallel
INFO : Launching Job 2 out of 15
INFO : Starting task [Stage-15:MAPRED] in parallel
INFO : Launching Job 3 out of 15
INFO : Starting task [Stage-16:MAPRED] in parallel
INFO : Starting task [Stage-27:CONDITIONAL] in parallel
INFO : Starting task [Stage-32:MAPREDLOCAL] in serial mode
INFO : Execution completed successfully
INFO : MapredLocal task succeeded
INFO : Starting task [Stage-25:CONDITIONAL] in parallel
INFO : Launching Job 5 out of 15
INFO : Starting task [Stage-26:MAPRED] in parallel
INFO : Launching Job 6 out of 15
INFO : Starting task [Stage-2:MAPRED] in parallel
INFO : Starting task [Stage-22:CONDITIONAL] in parallel
INFO : Launching Job 7 out of 15
INFO : Starting task [Stage-3:MAPRED] in parallel
INFO : Launching Job 8 out of 15
INFO : Starting task [Stage-4:MAPRED] in parallel
INFO : Launching Job 9 out of 15
INFO : Starting task [Stage-5:MAPRED] in parallel
INFO : Starting task [Stage-20:CONDITIONAL] in parallel
INFO : Launching Job 10 out of 15
INFO : Starting task [Stage-6:MAPRED] in parallel
INFO : Launching Job 11 out of 15
INFO : Starting task [Stage-7:MAPRED] in parallel
INFO : Starting task [Stage-13:CONDITIONAL] in serial mode
INFO : Stage-10 is selected by condition resolver.
INFO : Stage-9 is filtered out by condition resolver.
INFO : Stage-11 is filtered out by condition resolver.
INFO : Starting task [Stage-10:MOVE] in serial mode
INFO : Moving data to directory hdfs://dh01:8020/user/hive/warehouse/tdxdb.db/t_group_ent_info/.hive-staging_hive_2024-06-28_11-11-52_726_3253104069358746967-46/-ext-10000 from hdfs://dh01:8020/user/hive/warehouse/tdxdb.db/t_group_ent_info/.hive-staging_hive_2024-06-28_11-11-52_726_3253104069358746967-46/-ext-10008
INFO : Starting task [Stage-0:MOVE] in serial mode
INFO : Loading data to table tdxdb.t_group_ent_info from hdfs://dh01:8020/user/hive/warehouse/tdxdb.db/t_group_ent_info/.hive-staging_hive_2024-06-28_11-11-52_726_3253104069358746967-46/-ext-10000
INFO : Starting task [Stage-8:STATS] in serial mode
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-16: Map: 2 Reduce: 1 Cumulative CPU: 21.78 sec HDFS Read: 30826969 HDFS Write: 1230069 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-15: Map: 9 Reduce: 9 Cumulative CPU: 145.46 sec HDFS Read: 377189780 HDFS Write: 218441824 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-1: Map: 9 Reduce: 9 Cumulative CPU: 145.27 sec HDFS Read: 446904283 HDFS Write: 411352262 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-26: Map: 64 Cumulative CPU: 3235.28 sec HDFS Read: 14113867094 HDFS Write: 30291640119 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-2: Map: 68 Reduce: 432 Cumulative CPU: 3882.94 sec HDFS Read: 7760741455 HDFS Write: 94094193 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-3: Map: 8 Reduce: 2 Cumulative CPU: 92.13 sec HDFS Read: 312683077 HDFS Write: 11310877 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-4: Map: 2 Reduce: 1 Cumulative CPU: 18.65 sec HDFS Read: 11320365 HDFS Write: 11310781 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 10.96 sec HDFS Read: 11317422 HDFS Write: 1708107 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-6: Map: 105 Reduce: 113 Cumulative CPU: 3787.39 sec HDFS Read: 30301077693 HDFS Write: 3614603 HDFS EC Read: 0 SUCCESS
INFO : Stage-Stage-7: Map: 4 Reduce: 1 Cumulative CPU: 29.38 sec HDFS Read: 3664294 HDFS Write: 1505025 HDFS EC Read: 0 SUCCESS
INFO : Total MapReduce CPU Time Spent: 0 days 3 hours 9 minutes 29 seconds 240 msec
INFO : Completed executing command(queryId=hive_20240628111152_79f7458b-5497-4b76-803a-560abca802e5); Time taken: 795.243 seconds
INFO : OK
17,472 rows affected (796.499 seconds)