hive--map数量过小导致hive运行缓慢

已于 2023-06-23 06:25:49 修改
阅读量2.4k 收藏 4
点赞数 1
分类专栏： 错误集锦 文章标签： hive
于 2021-03-11 15:16:09 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_46893497/article/details/113846400
版权

map数量过小导致hive运行缓慢

  • 网页显示
    在这里插入图片描述
  • 日志
    • Not uberizing job_1607937945101_247633 because: not enabled; too much RAM;
    • nput size for job job_1607937945101_246670 = 38627311. Number of splits = 1
    • Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0这条一直出现。。。。。。。
  • 思路
    • 数据量过大,map数过少
    • 调整 参数
      • set mapred.max.split.size=25000000;
      • set mapred.min.split.size.per.node=25000000;
      • set mapred.min.split.size.per.rack=25000000;
    • 再次查看日志 Number of splits = 4，速度大大加快
    • 如果不行（则应该是压缩不可切分造成）
  • 完整日志可以参考下面

Logged in as: yarn
Application
About
Jobs
Tools
Log Type: syslog

Log Upload Time: 星期一 三月 01 18:18:41 +0800 2021

Log Length: 169813

2021-03-01 16:57:51,172 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1607937945101_246670_000001
2021-03-01 16:57:51,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2021-03-01 16:57:51,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33bc72d1)
2021-03-01 16:57:51,604 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2021-03-01 16:57:51,606 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2021-03-01 16:57:52,081 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-01 16:57:52,215 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2021-03-01 16:57:52,216 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2021-03-01 16:57:52,216 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2021-03-01 16:57:52,217 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2021-03-01 16:57:52,217 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2021-03-01 16:57:52,221 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2021-03-01 16:57:52,221 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2021-03-01 16:57:52,222 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2021-03-01 16:57:52,257 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,277 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,293 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:52,306 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2021-03-01 16:57:52,340 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2021-03-01 16:57:52,536 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2021-03-01 16:57:52,591 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2021-03-01 16:57:52,591 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2021-03-01 16:57:52,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1607937945101_246670 to jobTokenSecretManager
2021-03-01 16:57:52,709 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1607937945101_246670 because: not enabled; too much RAM;
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1607937945101_246670 = 38627311. Number of splits = 1
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1607937945101_246670 = 0
2021-03-01 16:57:52,729 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from NEW to INITED
2021-03-01 16:57:52,730 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1607937945101_246670.
2021-03-01 16:57:52,753 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2021-03-01 16:57:52,762 INFO [Socket Reader #1 for port 36866] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 36866
2021-03-01 16:57:52,793 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2021-03-01 16:57:52,826 INFO [IPC Server listener on 36866] org.apache.hadoop.ipc.Server: IPC Server listener on 36866: starting
2021-03-01 16:57:52,826 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-03-01 16:57:52,828 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdhuatdn62-209.com/10.9.62.209:36866
2021-03-01 16:57:52,883 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2021-03-01 16:57:52,889 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2021-03-01 16:57:52,893 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2021-03-01 16:57:52,902 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2021-03-01 16:57:52,932 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2021-03-01 16:57:52,932 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2021-03-01 16:57:52,934 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2021-03-01 16:57:52,934 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2021-03-01 16:57:52,941 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 46085
2021-03-01 16:57:52,941 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2021-03-01 16:57:52,964 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/jars/hadoop-yarn-common-2.6.0-cdh5.13.3.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_46085_mapreduce____.frd8sy/webapp
2021-03-01 16:57:53,196 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:46085
2021-03-01 16:57:53,196 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 46085
2021-03-01 16:57:53,435 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2021-03-01 16:57:53,439 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1607937945101_246670
2021-03-01 16:57:53,440 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2021-03-01 16:57:53,441 INFO [Socket Reader #1 for port 43229] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43229
2021-03-01 16:57:53,481 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2021-03-01 16:57:53,481 INFO [IPC Server listener on 43229] org.apache.hadoop.ipc.Server: IPC Server listener on 43229: starting
2021-03-01 16:57:53,501 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2021-03-01 16:57:53,502 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2021-03-01 16:57:53,502 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2021-03-01 16:57:53,564 INFO [main] org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider: Failing over to rm709
2021-03-01 16:57:53,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:131072, vCores:32>
2021-03-01 16:57:53,600 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.w_hanzhiying
2021-03-01 16:57:53,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2021-03-01 16:57:53,603 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2021-03-01 16:57:53,612 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from INITED to SETUP
2021-03-01 16:57:53,615 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2021-03-01 16:57:53,617 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from SETUP to RUNNING
2021-03-01 16:57:53,752 INFO [Socket Reader #1 for port 36866] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hive (auth:SIMPLE) for protocol=interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
2021-03-01 16:57:53,772 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1607937945101_246670, File: hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1.jhist
2021-03-01 16:57:53,791 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from NEW to SCHEDULED
2021-03-01 16:57:53,792 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2021-03-01 16:57:53,793 INFO [Thread-53] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:2048, vCores:1>
2021-03-01 16:57:53,904 INFO [IPC Server handler 0 on 36866] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP   job_1607937945101_246670. Report-size will be 1
2021-03-01 16:57:53,964 INFO [IPC Server handler 0 on 36866] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE   job_1607937945101_246670. Report-size will be 0
2021-03-01 16:57:53,983 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://lhService01:8020]
2021-03-01 16:57:54,603 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2021-03-01 16:57:54,632 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1607937945101_246670: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:179712, vCores:199> knownNMs=7
2021-03-01 16:57:55,642 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
2021-03-01 16:57:55,671 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e64_1607937945101_246670_01_000002 to attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,673 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 16:57:55,743 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://lhService01/user/hive/.staging/job_1607937945101_246670/job.jar
2021-03-01 16:57:55,746 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hive/.staging/job_1607937945101_246670/job.xml
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
2021-03-01 16:57:55,753 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2021-03-01 16:57:55,784 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx1639m
2021-03-01 16:57:55,786 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2021-03-01 16:57:55,791 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e64_1607937945101_246670_01_000002 taskAttempt attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,794 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1607937945101_246670_m_000000_0
2021-03-01 16:57:55,855 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1607937945101_246670_m_000000_0 : 13562
2021-03-01 16:57:55,858 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1607937945101_246670_m_000000_0] using containerId: [container_e64_1607937945101_246670_01_000002 on NM: [cdhuatdn62-209.com:8041]
2021-03-01 16:57:55,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2021-03-01 16:57:55,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from SCHEDULED to RUNNING
2021-03-01 16:57:56,675 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1607937945101_246670: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:177664, vCores:194> knownNMs=7
2021-03-01 16:57:57,308 INFO [Socket Reader #1 for port 43229] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1607937945101_246670 (auth:SIMPLE)
2021-03-01 16:57:57,317 INFO [Socket Reader #1 for port 43229] SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for job_1607937945101_246670 (auth:TOKEN) for protocol=interface org.apache.hadoop.mapred.TaskUmbilicalProtocol
2021-03-01 16:57:57,327 INFO [IPC Server handler 1 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1607937945101_246670_m_70368744177666 asked for a task
2021-03-01 16:57:57,327 INFO [IPC Server handler 1 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1607937945101_246670_m_70368744177666 given task: attempt_1607937945101_246670_m_000000_0
2021-03-01 16:58:10,060 INFO [IPC Server handler 6 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0
2021-03-01 16:58:16,148 INFO [IPC Server handler 12 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0


此处省略

org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 0.0
2021-03-01 18:18:33,300 INFO [IPC Server handler 15 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1607937945101_246670_m_000000_0 is : 1.0
2021-03-01 18:18:33,305 INFO [IPC Server handler 26 on 43229] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:33,309 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from RUNNING to SUCCESS_FINISHING_CONTAINER
2021-03-01 18:18:33,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:33,348 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1607937945101_246670_m_000000 Task Transitioned from RUNNING to SUCCEEDED
2021-03-01 18:18:33,351 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2021-03-01 18:18:33,352 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from RUNNING to COMMITTING
2021-03-01 18:18:33,362 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT
2021-03-01 18:18:33,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 
2021-03-01 18:18:33,406 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1607937945101_246670Job Transitioned from COMMITTING to SUCCEEDED
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2021-03-01 18:18:33,423 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2021-03-01 18:18:33,426 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
2021-03-01 18:18:33,794 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 18:18:34,090 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1.jhist to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp
2021-03-01 18:18:34,215 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp
2021-03-01 18:18:34,222 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://lhService01:8020/user/hive/.staging/job_1607937945101_246670/job_1607937945101_246670_1_conf.xml to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp
2021-03-01 18:18:34,307 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp
2021-03-01 18:18:34,317 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670.summary_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670.summary
2021-03-01 18:18:34,319 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670_conf.xml
2021-03-01 18:18:34,321 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist_tmp to hdfs://lhService01:8020/user/history/done_intermediate/hive/job_1607937945101_246670-1614589069421-hive-hv_app_a09_longfor_gydtsy_poi_20210301164855_13328-1614593913403-1-0-SUCCEEDED-root.users.w_hanzhiying-1614589073606.jhist
2021-03-01 18:18:34,321 INFO [Thread-298] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2021-03-01 18:18:34,322 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1607937945101_246670_m_000000_0
2021-03-01 18:18:34,359 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1607937945101_246670_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2021-03-01 18:18:34,364 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 
2021-03-01 18:18:34,365 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://cdhuatmaster62-201.com:19888/jobhistory/job/job_1607937945101_246670
2021-03-01 18:18:34,374 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2021-03-01 18:18:35,375 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2021-03-01 18:18:35,376 INFO [Thread-298] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://lhService01 /user/hive/.staging/job_1607937945101_246670
2021-03-01 18:18:35,388 INFO [Thread-298] org.apache.hadoop.ipc.Server: Stopping server on 43229
2021-03-01 18:18:35,392 INFO [IPC Server listener on 43229] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 43229
2021-03-01 18:18:35,392 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2021-03-01 18:18:35,394 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
2021-03-01 18:18:35,396 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2021-03-01 18:18:40,394 INFO [Thread-298] org.apache.hadoop.ipc.Server: Stopping server on 36866
2021-03-01 18:18:40,394 INFO [IPC Server listener on 36866] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 36866
2021-03-01 18:18:40,394 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2021-03-01 18:18:40,396 INFO [Thread-298] org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
韩家小志
关注
专栏目录
Hive学习笔记 ---- 支持Update和Delete以及Merge
杨鑫newlife的专栏
03-18 2455
在最早2013年接触Hive的时候就总看到各种材料说着Hive有一个很大的弊端，就是分布式系统依托底层HDFS存储系统 没法对行级别的数据进行update更新和delete行级别的删除和merge行级别合并操作。 但是后来隐约的听说Hive又可以支持update和delete了，我就去做了一个调研，这里是总结的说明文档。 其中官网的WIKI中记录，开始支持行级别的update和d...
hive 很慢解决办法
热门推荐
暮雨听轩的专栏
04-09 1万+
hive 很慢：运行一下一下语句：set mapred.child.java.opts=-Xmx1024m;set mapreduce.map.java.opts=-Xmx1310m;set yarn.app.mapreduce.am.command-opts=-Xmx2457m;set mapreduce.reduce.java.opts=-Xmx2620m;...
Hive map阶段缓慢，优化过程详细分析
weixin_33697898的博客
08-01 7192
背景 同事写了这样一段HQL(涉及公司数据，表名由假名替换，语句与真实场景略有不同，但不影响分析)： CREATETABLEtmpAS SELECT t1.exk, t1.exv, M.makenameASm_makename, S.makenameASs_maken...
Hive map阶段缓慢
hellojoy的博客
01-24 4111
不同数据类型关联产生数据倾斜 场景：用户表中user_id字段为int，log表中user_id字段既有string类型也有int类型。当按照user_id进行两个表的Join操作时，默认的Hash操作会按int型的id来进行分配，这样会导致所有string类型id的记录都分配到一个Reducer中。 解决方法：把数字类型转换成字符串类型 select * from users a   l...
hive 分配map数过少导致任务执行慢
weixin_30701521的博客
08-24 500
数据表大概150M，但是只有几个字段，导致行数特别多，当使用正则表达式去匹配时执行较慢。 解决思路：增大map数; //设置reduce数为150，将原表分成150份，map数无法直接设置，因为和输入文件数和文件大小等几个参数决定set mapred.reduce.tasks = 150; //在map完成阶段不对文件进行合并，相应还有个mapredfiles，是在整个任务完成后不对输出文...
Hive性能优化--map数和reduce数
a75543332的博客
04-17 125
转自http://superlxw1234.iteye.com/blog/1582880 一、 控制hive任务中的map数:1. 通常情况下，作业会通过input的目录产生一个或者多个map任务。主要的决定因素有： input的文件总个数，input的文件大小，集群设置的文件块大小(目前为128M, 可在hive中通过set dfs.block.size;命令查...
hive任务，一些map长时间无法结束
王某的博客
08-20 5072
多线程同步写ES 切分文件 在这篇文章中使用代码切割文件，这里尝试使用split命令切分文件。 split -l 250000 all_rate.csv 25万行为分界线分割文件 import org.elasticsearch.action.bulk.BulkRequestBuilder; import org.elasticsearch.client.transport.Tra...
Hive高级用法01-hive调优
小梁
07-24 1821
文章目录1. 查看执行计划    1.1 执行计划有什么用:    1.2 怎么查询执行计划    1.3 简单示例2.建表优化      2.1 建分区表         2.1.1 什么是分区表   &nbs
Hive优化（提高hive运行速度）
徐磊的博客
02-18 1万+
一、Hive使用本地模式执行操作（Hive使用资源大于20M的时候还是会采用集群yarn的方式运行） 将hive的job交给yarn执行太慢，hive支持本地模式 設置本地模式之前的，耗時 9.068 設置本地模式之后的，耗時 0.29 设置hive基于本地模式的方法 1、设置临时的 在每次执行hql语句之前 都要先执行下面这句话 set hive.exec.mode.local.auto...
大数据--hive4--HiveSQL查询语法总结
数据挖掘+大数据研发+算法学习
05-03 1202
目录 一：Hive SQL与SQL的区别 二：Hive表关联条件不支持不等值连接 三：桶或者簇 四：group by 数据分组 五：collect_list和collect_set用法 六：根据Hive Sql语句的执行顺序而进行的优化 七：SQL的join操作 八： UNION和 union all 数据拼接 九：其他 一：Hive SQL与SQL的区别 1）用法上的区别： HQL不支持行级别的增删改，所有数据加载的时候就已经确定，不可更改。 不支持事务 支持分区存储 2）h.
spark sql 查询hive task过多导致任务执行缓慢
weixin_43828782的博客
08-21 2501
一、问题 跑了一年多的spark sql任务突然挂掉了，查看yarn的界面，任务并行度达到了恐怖的81835个,运行速度自然也快不到哪去。spark任务界面如下图所示 二、解决 1.排查过程 sql类似于下面这段sql,开始以为是先join再where过滤导致数据量特别大导致task过多(想法是错误的后面验证了是先过滤再进行join的) select * from A join B on A.c=B.c WHERE A.date=2020-08-20 and B.date=2020-08-20 想法1:改
大数据开发之Hive案例篇4-Map数过少导致性能低下
只是甲的博客
10-20 576
大数据 Hive 非等值连接 Map数过少 参数调整
怎样去找出线上hive或mapreduce运行慢的原因
学无止境
08-19 9690
hive mapreduce 都会转成java。所以要用的java分析工具。当看到map或reduce 阶段运行缓慢，我们去线上找到mapreduce的进程： 1. top 发现进程： 2. top -Hp 32486 找到进程中cpu或mem磁盘消耗最高的线程： 3.  用jstack主要用来查看某个Java进程内的线程堆栈信息： bin/jstack -F 26510
MapReduce跑得慢的原因以及优化方法
weixin_44976835的博客
11-04 3392
MapReduce跑得慢的原因 MapReduce程序的效率瓶颈在于两个方面： 一、 计算机性能 CPU、内存、磁盘健康、网络 二、 I/O操作 数据倾斜 map和reduce数量设置不合理 reduce等待时间过久 小文件过多 大量不可拆分的超大文件 spill次数过多 merge次数过多 我们优化的重点是I/O操作 MapReduce优化方法 一、数据输入 合并小文件：在执行mr任务前将小文件进行合并，因为大量的小文件会产生大量的map任务，增大map任务装载次数，而任务的装载比较耗时，从而导
Hive性能调优行之有效的优化方法
大模型大数据攻城狮的专栏
09-19 400
这种特定的 Hive 查询调优最佳实践可帮助您了解您的代码如何工作，以便在您将大数据集投入其中之前获得所需的结果。由于需要输入、输出或存储在内存中的数据更少，因此这改进了跨时间和效率向量的查询。如果您想消除在运行时从多个表连接数据的需要，Hive 专家建议将数据反规范化作为一种​​首选的 Hive 性能调整方法。嵌套连接的执行成本也很高。分区是一种常见的 Hive 查询调优策略，它根据键将表数据放置在表位置的单独子目录中。分区键提供了一个机会来定位表数据的一个子集，而不是扫描您的操作不需要的数据。
Hive SQL作业执行慢，怎么解决
m0_70220205的博客
11-02 931
4，a left join b 可以转换为：b left join a 加 union，基于维度对指标进行。：需要过滤链接条件中的NULL值：解决方法1：NULL不参与关联，解决方法2：给空值分配随机的key值。实现广播的小表链接大表，来提升链接的效率。1，可以舍弃热点数据，稀疏的key数据与密集的key数据分开计算，避免。小文件过多造成执行过程中的拉取文件的大量随机读，带来任务延迟。3，大表热点数据在key上拼接0～n的前缀/后缀，同时。，限制分区，过滤冗余数据，拆解任务分批次执行。
分布式系统MaxCompute/Hadoop日志分析与优化流程
Monkeyhou1992的博客
03-24 612
很久没有认真复盘一下，接着两年前的文章Hadoop性能优化概述-数据膨胀&数据倾斜系统性总结一下优化方案和如何从日志中分析性能卡点，同时方便自己加深印象0.0~我们在提交作业后，一般会分为4各阶段：（1）预处理阶段；（2）编译阶段；（3）执行阶段；（4）结果返回。其中，预处理阶段除了语法、调度配置与参数配置等出现问题外，一般不会成为卡点。我们重点关注复杂任务的编译、执行阶段的优化以及性能卡点。
Hive性能优化
La victoria的博客
11-30 424
Hive性能优化 数据量大不是问题，数据倾斜是个问题。 jobs数比较多的作业运行效率相对比较低，比如即使有几百行的表，如果多次关联多次汇总，产生十几个jobs，耗时很长。原因是map reduce作业初始化的时间是比较长的。 sum,count,max,min等UDAF，不怕数据倾斜问题,hadoop在map端的汇总合并优化，使数据倾斜不成问题。 count(d...
hive -e和hive -f的区别
最新发布
11-22
hive -e和hive -f的区别如下： 1. hive -e：使用-e选项可以在命令行中直接执行短命令。例如，可以使用以下命令在Hive中执行查询： ``` hive -e "use tv; select * from test;" ``` 这将在Hive中执行两个操作：首先，使用tv数据库，然后从test表中选择所有的数据。 2. hive -f：使用-f选项可以执行一个包含Hive查询的文件。例如，可以使用以下命令执行名为test.hql的Hive查询文件： ``` hive -f test.hql ``` 这将执行test.hql文件中的所有Hive查询。 总结起来，hive -e用于执行短命令，而hive -f用于执行包含Hive查询的文件。
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值