最近hive在运行的过程中,出现CPU 100%的情况还有,但是间隔时间较长了。
之前出现这种情况是因为NIO的那个异常导致,如果中间空闲间隔较长,在这段时间内CPU会出现100%的情况。
而现在出现这种情况,跟NIO异常无关,我已经在日志查找不到发送失败的日志了,而该100%的占用基本不会停止,只能强行kill掉,根据经验判断,一般CPU100%的情况多少是由于无限循环导致的(如果不信,大家可以自己在本机进行测试看看,就会发现无限循环时,查看CPU的使用率情况)。
经在网上查找发现一个关于此问题的描述,截取如下:
[Hive-dev] 100% CPU usage of Hive
Hi all,
I encounter a problem when running some HQL(under CLI).
The hive process consumes full core CPU usage and turns out never ending
that query.
I use jtask to get the java stack and finding out it may be a bug cause
by concurrent modification of HashMap.
The stack trace looks this:
*"Thread-1465" prio=10 tid=0x00002aaabc0a7000 nid=0x5be0 runnable
[0x00000000431ee000]*
* java.lang.Thread.State: RUNNABLE*
* at java.util.HashMap.transfer(HashMap.java:484)*
* at java.util.HashMap.resize(HashMap.java:463)*
* at java.util.HashMap.addEntry(HashMap.java:755)*
* at java.util.HashMap.put(HashMap.java:385)*
* at java.util.HashSet.add(HashSet.java:200)*
* at org.apache.hadoop.hive.ql.QueryPlan.extractCounters(QueryPlan.java:357)
*
* at org.apache.hadoop.hive.ql.QueryPlan.getQueryPlan(QueryPlan.java:422)*
* at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:595)*
* at
org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:496)
*
* at
org.apache.hadoop.hive.ql.exec.ExecDriver.logPlanProgress(ExecDriver.java:1009)
*
* at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:359)
*
* at
org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:685)
*
* at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:458)*
* at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)*
* at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)*
* at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)*
* at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)*
The "Thread-1465" is the only running thread except for some other JVM
thread like Finalizer.
Environment:
Hive 0.8.1.
Hadoop 1.0.0
Sun Java 1.6.0_23
The attachment is the full stack trace.
根据此日志的描述,是在hashMap这个类中出现的问题,我自己通过源码也能发现在此方法中有个循环语句(do...while)。为我的猜测提供了有利的依据,不过为了更加证实是否是此方法出现的问题,我将调整日志级别,和增加LOG的方式进行验证。
方法截图如下:
/**
* Transfers all entries from current table to newTable.
*/
void transfer(Entry[] newTable) {
Entry[] src = table;
int newCapacity = newTable.length;
for (int j = 0; j < src.length; j++) {
Entry<K,V> e = src[j];
if (e != null) {
src[j] = null;
do {
Entry<K,V> next = e.next;
int i = indexFor(e.hash, newCapacity);
e.next = newTable[i];
newTable[i] = e;
e = next;
} while (e != null);
}
}
}
另外就是当hive进程的占用率是100%时,我检查过hadoop的job运行情况,没有任何job在运行,说明hive脚本在运行时,没有job提交到hadoop的Jobtracker这边。
后续将持续关注此问题…………………………