hadoop装mysql配置hive神坑集合

真的是一步一个坑啊！！！！

 

初始化数据库

sudo mysql_secure_installation

 

 

 

 

 

 

 

 

 

 

 

 

这条赋权命令分三部分写

grant all on *.* to hive@localhost identified by 'hive';

 

 

 

 

 

 

 

 

 

 

 

 

 

 

创建两个文件输入内容

 

 

 

 

 

 

问题一，multiple SLF4J bindings

 

 

 

 

 

hive 和 hadoop 中的 SLF4J 重复了

在两个中的位置分别如下：

- /usr/local/hive/lib/

- /usr/local/hadoop/share/hadoop/common/lib/

解决办法：

 

删除任意一个 log4j-slf4j-impl.jar 即可，

建议保留版本高版本。

 

删除任意一个jar包即可，建议 保留高版本的

把hive下的那个删了

问题2.

Exception in thread" main" java. Lang . NoSuchMethodError: 。。。。

 

Hadoop的和hive Guava的版本不一致保留复制最高

 

cp /usr/Local/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar /usr/local/hive/lib

 

问题3.拒绝连接

 

错误原因：

 

1.没开hadoop

2.配置文件错误，连接mysql的配置以及hadoop的配置错误

 

 

问题4.多处语句错误

 

 

原因：版本、配置不一致。

解决方法 ：使用 schematool 工具。 Hive 现在包含一个用于 Hive Metastore 架构操控的脱机工具，名为 schematool. 此工具可用于初始化当前 Hive 版本的 Metastore 架构。此外，其还可处理从较旧版本到新版本的架构升级

 

cd /usr/local/hive/bin
./schematool -initSchema -dbType mysql

 

 

 

 

 

问题5.Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

 

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException:  Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

　 参考： http://blog.csdn.net/l1028386804/article/details/51566262

　 解决办法：

　　进入所安装的 Hive 的 conf 目录，找到 hive-site.xml ，（若没修改，则是 hive-default.xml.template ）。

<property>

  <name>hive.metastore.schema.verification</name>

  <value>true</value>

   <description>

   Enforce metastore schema version consistency.

   True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic

         schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures

         proper metastore schema migration. (Default)

   False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.

   </description>

</property>

 

 

改为

<property>

  <name>hive.metastore.schema.verification</name>

  <value> false </value>

   <description>

   Enforce metastore schema version consistency.

   True: Verify that version information stored in metastore matches with one from Hive jars.  Also disable automatic

         schema migration attempt. Users are required to manully migrate schema after Hive upgrade which ensures

         proper metastore schema migration. (Default)

   False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.

   </description>

</property>

 

　　然后，再重启 hive ，进入即可。

或者

首先进入hive/bin目录下

执行命令    ./hive --service metastore &

 　　

 

 

 

 

 

 

 

 

问题6:Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

[hadoop@HadoopSlave1 hive-1.2.1]$ bin/hive

 

Logging initialized using configuration in jar:file:/home/hadoop/app/hive-1.2.1/lib/hive-common-1.2.1.jar!/hive-log4j.properties

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)

at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)

at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)

at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)

at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)

at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)

... 8 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)

... 14 more

Caused by: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: hfds)

at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:1213)

at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:106)

at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:140)

at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:146)

at org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159)

at org.apache.hadoop.hive.metastore.Warehouse.getDefaultDatabasePath(Warehouse.java:177)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB_core(HiveMetaStore.java:600)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)

at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)

at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)

at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)

at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)

at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)

... 19 more

[hadoop@HadoopSlave1 hive-1.2.1]$

 

将hadoop中share/hadoop/yarn/lib路径下jline包换成hive中匹配的jar包

 

 

 

 

 

 

问题7.启动运行hive在ctl输出过多info

配置好hive后，输入hive命令启动时会出现许多INFO和WARN信息。如下： 至今仍未解决

 

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

15/04/21 11:31:45 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative

15/04/21 11:31:45 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect.  Use hive.hmshandler.retry.* instead

Logging initialized using configuration in file:/home/hbase/install/hive-0.12.0-cdh5.1.0/conf/hive-log4j.properties

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/hbase/install/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/home/hbase/install/hive-0.12.0-cdh5.1.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

 

原因是在新的Hadoop版本中，很多参数都废弃了，所以这些信息一定会显示出来。解决的办法是在$HADOOP_HOME/etc/hadoop/log4j.properties中将 log4j.logger.org.apache.hadoop.conf.Configuration.deprecation打开。如下：

# Uncomment the following line to turn off configuration deprecation warnings.

log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN

 

log4j.properties修改为：hive.root.logger=WARN,console 即可

 

或者   在 hive 安装目录下的 bin 目录下， ./hive -hiveconf hive.root.logger=DEBUG,console

 

 

问题8.词频统计找不到文件

 

 

 

分析原因：

 

When using the JDBC driver, the command executes on the HiveServer2 side. The file is evaluated to locally exist on the server, which is not true in your case (it exists on the local client program machine).

 

Try instead to load the file to HDFS first, and use a HDFS URI in the LOAD DATA statement to make the server find it.

从上面的解释可知， hive导入数据语句  load data [local] inpath ，是一个服务器端的指令，它是在服务器端执行。因此指定local时表明加载的文件为本地文件，但是这里的local，在hive中指的是 hiveserver 服务所在的机器，而不是hivecli 或 beeline客户端所在的机器（生产环境大都是 hiveserver 和 hivecli不在同一个机器）。

 

总结：

 

    解决方法一：把要加载的文件上传到 hiveserver所在的服务器(这一般是不可能的事情)，然后执行  load data local inpath [path] [overwrite] into table table_name.

 

    解决方法二:先将本地文件上传至hdfs，然后使用 load data inpath [hdfspath] [overwrite] into table table_name.

 

方法二为推荐做法。