Hive0.13+Hadoop cdh3中的log4j问题

Hive0.12+ 在Hadoop cdh3（1.0）中使用，会有如下错误：

 

log4j:ERROR Could not instantiate class [org.apache.hadoop.hive.shims.HiveEventCounter].
java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.log.metrics.EventCounter
        at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:123)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:115)
        at org.apache.hadoop.hive.shims.ShimLoader.getEventCounter(ShimLoader.java:98)
        at org.apache.hadoop.hive.shims.HiveEventCounter.<init>(HiveEventCounter.java:34)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at java.lang.Class.newInstance0(Class.java:355)
        at java.lang.Class.newInstance(Class.java:308)
        at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:330)
        at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:121)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:664)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:647)
        at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:544)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:440)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:476)
        at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:354)
        at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:127)
        at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:77)
        at org.apache.hadoop.hive.common.LogUtils.initHiveExecLog4j(LogUtils.java:69)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.setupChildLog4j(ExecDriver.java:611)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:681)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.log.metrics.EventCounter
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:120)
        ... 27 more
log4j:ERROR Could not instantiate appender named "EventCounter".

 

 

原因是hive-log4j.properties中log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter

的配置，因为Hadoop的classpath中并没有这个类，因此报错。

 

解决办法：

1.将hive-exec-0.13.1.jar拷贝到各个Hadoop节点$HADOOP_HOME/lib/目录下，重启集群；

   这个方法显然不大合适；

2.将hive-log4j.properties中的log4j.appender.EventCounter值改为org.apache.hadoop.metrics.jvm.EventCounter，因为$HADOOP_HOME/conf/log4j.properties中也是这样配置的。

 

 

另外，按照方法2修改之后，使用hive普通查询没问题，在MapJoin的时候，又报出这个错。

原因是MapJoin会启动一个Hadoop LocalTask，使用命令：

 

$HADOOP_HOME/bin/hadoop jar $HIVE_HOME/lib/hive-exec-0.13.1.jar org.apache.hadoop.hive.ql.exec.mr.ExecDriver -localtask -plan file:/tmp/plan.xml   -jobconffile file:/tmp/jobconf.xml

该local task中使用了hive-exec-0.13.1.jar里面自带的hive-log4j.properties，其中的log4j.appender.EventCounter仍然是org.apache.hadoop.hive.shims.HiveEventCounter，因此又报错。

 

解决办法： 修改hive-exec-0.13.1.jar中的hive-log4j.properties，将log4j.appender.EventCounter值改为org.apache.hadoop.metrics.jvm.EventCounter

 

之后，使用MapJoin也没问题。