目录
0. 安装背景:
根据林子雨老师的Hive3.1.2安装进行安装,发现部分步骤及设置由于新版本原因需要更新。
我安装的版本为
Hadoop -3.3.6
Hive -3.1.3
mysql-connector -8.3.0
1. 配置hive-site.xml:
林子雨老师的配置
解释:
1.1 第一条红线:
为jdbc驱动名称,用于MySQL和Hive的连接。林子雨老师的驱动名称为旧版本,运行会出现如下报错。
Loading class `com.mysql.jdbc.Driver'. This is deprecated.
The new driver class is `com.mysql.cj.jdbc.Driver'.
The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
1.1.1解释:
警告是关于MySQL JDBC驱动程序的类名已更改的问题。新的MySQL JDBC驱动程序类名为com.mysql.cj.jdbc.Driver
,而不再是com.mysql.jdbc.Driver
。警告指出新的驱动程序类通过SPI(Service Provider Interface)自动注册,因此通常不需要手动加载驱动程序类。
1.1.2解决办法:
若使用的是官网下载的(注意官网下载的驱动包,解压后的jar文件位置不再和林子雨老师示范的位置一致,需要到解压文件后/usr/share/java/中找到对应的jar包),需要将第一处红线修改为com.mysql.cj.jdbc.Driver
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
1.2 第二、三条红线:
分别为访问Hive的用户名和对应用户密码,用于后续的用户访问Hive。即对应林子雨老师该步骤,第一个Hive为用户名,第二个为密码。
若在hive-site.xml中配置用户名为“user1”,访问密码为“123”,则MySQL需使用如下命令:
grant all on *.* to user1@localhost identified by '123';
若出现错误:You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near,则改为下面命令:
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';
GRANT ALL ON *.* TO 'hive'@'localhost';
1.3 完整hive-site.xml配置文件:
注意最后还添加了metastore.schema初始化配置信息,用于后续2.1.2初始化使用
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description> Enforce metastore schema version consistency. True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures proper metastore schema migration. (Default); False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
</property>
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hadoop</value>
<description>username to use against metastore database</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>7410</value>
<description>password to use against metastore database</description>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description> Enforce metastore schema version consistency. True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures proper metastore schema migration. (Default); False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
</property>
</configuration>
2. 安装重要补充
很遗憾,你在按照林子雨老师的配置完成后,可以启动Hive,但是却没法通过任何Hive指令对数据库进行访问(也能是可能作者太菜了,漏了啥),错误如下:
SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
该代码意为:无法实例化org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
,这可能是由于元数据存储(Metastore)的配置问题或其他配置错误导致的。
2.1 解决:
2.1.1启动客户端及元数据
出现上述问题的直接原因为metadata服务端未正常开启导致的,首先尝试能否启动,在terminal中直接输入以下代码尝试启动,注意在主目录下运行
hive --service metastore &
若输入以上命令出现以下问题,那么很遗憾,客户端没能完成启动,可能由于你设置的mysql元数据存储库没有进行schema初始化,接着进行第二步骤,若能完成启动,即可使用。
部分错误:
MetaException(message:Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8672)
at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8667)
at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8937)
at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8854)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:328)
at org.apache.hadoop.util.RunJar.main(RunJar.java:241)
Caused by: MetaException(message:Error creating transactional connection factory)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
... 11 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
2.1.2对元数据库进行schema初始化
输入以下命令进行初始化(首先进入你Hive安装位置的bin文件夹)
cd /usr/local/hive/bin #你安装hive对应位置的bin文件夹
./schematool -dbType mysql -initSchema
有可能报如下错误:
作者的错误,是由于1.2中hive-site.xml用户设置未正确匹配MySQL访问用户所致。建议仔细检查hive-site.xml是否一致,若有调整则需要和MySQL设置匹配。
若出现如下信息,这表明元数据库初始化完成
测试Hive,没问题后重启,开始你的Hadoop痛苦之旅吧
show databases;
参考文章:
3、Hive错误: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.Session
4、org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version