其它项目组需要一套hive环境,原来想会很快搭好,没想到还颇费了些周折。这里记录一下。
hadoop用现成的脚本安装。
mysql也用现成的。
hive搭建的主要过程是按照这个教程做的:
http://www.cnblogs.com/zlslch/p/6700695.html,这个帖子虽然是安装1.0的,但安装2.0、3.0也好使。作者费心了。
修改hive-site.xml也参考了https://stackoverflow.com/questions/27099898/java-net-urisyntaxexception-when-starting-hive
也参考这篇文章做了初始化元数据:
schematool -dbType mysql -initSchema
schematool -dbType mysql -info
参考官方指南https://cwiki.apache.org/confluence/display/Hive/GettingStarted做了:
hadoop目录初始化:
$ $HADOOP_HOME/bin/hadoop fs -mkdir /tmp $ $HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /tmp $ $HADOOP_HOME/bin/hadoop fs -chmod g+w /user/hive/warehouse
建表、修改表、插入数据等也参照官方指南。
============================================================================
出现的问题:
启用debug日志: hive -hiveconf hive.root.logger=DEBUG,console)
运行简单的show databases命令报错:
2019-03-20T18:00:24,901 ERROR [08224acd-57c4-4e6f-bbf0-12c81aeef9c9 main] CliDriver: Failed with exception java.io.IOException:java.lang.RuntimeException: Error in configuring object
java.io.IOException: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:602)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:115)
at org.apache.hive.common.util.ReflectionUtil.setConf(ReflectionUtil.java:103)
at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:87)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:221)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:381)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:314)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:112)
... 22 more
Caused by: java.lang.IllegalArgumentException: Compression codec org.apache.hadoop.io.compress.Bzip2Codec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:179)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 27 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.io.compress.Bzip2Codec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2122)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132)
... 29 more
分析:
命令是正常执行了,但是结果不能正常显示。
搜了一下,网上基本没有报 Class org.apache.hadoop.io.compress.Bzip2Codec not found这个错的,有报lzo错的。
初步判断是配置文件有错。把core-site.xml中关于compress的配置全部去掉,问题照旧。
再判断是本地库编译有问题,确实当初编译的hadoop不支持Bzip2Codec。于是装了个centos编译源码,再重装hadoop,问题照旧。
再仔细分析错误,忽然发现Bzip2Codec这个类名与编译出来的类名不一样。编译出来的是BZip2Codec。然后修改core-site.xml中的:
<property>
<name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
重起hadoop后,问题照旧。
再忽然想起hbase中还有一份core-site.xml,修改完重起,OK,问题解决。
真是个惨痛的教训,一个字母大小写耗了好几天时间。