hadoop job 未跑满资源_hadoop map 任务一直不执行也不失败,重跑正常,经常出现?...

这篇博客主要讨论了一个Hadoop MapReduce作业在运行过程中出现的问题,即job未充分利用资源,map任务长时间处于RUNNING状态但不执行也不失败。通过分析日志,发现任务在分配后并未正常启动,可能存在调度或资源分配问题。尽管重新运行任务可以解决暂时问题,但了解并解决这类问题对于优化集群性能至关重要。
摘要由CSDN通过智能技术生成

日志如下:

2018-03-09 00:05:08,455 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1513173511959_439708_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING

2018-03-09 00:05:08,455 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1513173511959_439708_m_000000 Task Transitioned from SCHEDULED to RUNNING

2018-03-09 00:05:08,944 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1513173511959_439708: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<794624 vcores:194> knownNMs=24794624>

2018-03-09 00:05:09,962 INFO [Socket Reader #1 for port 37592] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1513173511959_439708 (auth:SIMPLE)

2018-03-09 00:05:10,004 INFO [IPC Server handler 0 on 37592] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1513173511959_439708_m_49478023249922 asked for a task

2018-03-09 00:05:10,005 INFO [IPC Server handler 0 on 37592] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1513173511959_439708_m_49478023249922 given task: attempt_1513173511959_439708_m_000000_0

2018-03-09 00:05:47,006 INFO [Socket Reader #1 for port 37592] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1513173511959_439708 (auth:SIMPLE)

2018-03-09 00:06:23,117 INFO [Socket Reader #1 for port 37592] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1513173511959_439708 (auth:SIMPLE)

2018-03-09 00:06:23,132 INFO [IPC Server handler 23 on 37592] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1513173511959_439708_m_000000_0 is : 1.0

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值