Hive本地模式安装配置

最新推荐文章于 2024-07-29 14:19:55 发布

EaShion1994

最新推荐文章于 2024-07-29 14:19:55 发布

阅读量2k

点赞数

分类专栏： Hadoop 文章标签：分布式 hive hadoop

本文链接：https://blog.csdn.net/EaShion1994/article/details/52056300

版权

Hadoop 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

准备：

hive安装包 http://mirrors.hust.edu.cn/apache/ 下载apache-hive-2.1.0-bin.tar.gz

mysql RPM压缩包 http://dev.mysql.com/downloads/mysql/5.5.html#downloads

基本步骤：

一. 卸载mysql

二. 安装mysql

三. 安装hive

一. 卸载mysql

#1.检查是否已经安装MySQL组件
[root@sst_search_a /]# rpm -qa | grep -i mysql
MySQL-devel-5.6.23-1.linux_glibc2.5
MySQL-client-5.6.23-1.linux_glibc2.5
MySQL-server-5.6.23-1.linux_glibc2.5

#2.卸载前关闭MySQL服务
[root@sst_search_a /]# service mysql status
 MySQL running (25673)[  OK  ]
[root@sst_search_a /]# service mysql stop
 Shutting down MySQL..[  OK  ]
[root@sst_search_a /]# service mysql status
 MySQL is not running[FAILED]

#3.收集MySQL对应文件夹信息
[root@sst_search_a /]# find / -name mysql
/etc/rc.d/init.d/mysql
/etc/logrotate.d/mysql
/var/lock/subsys/mysql
/var/lib/mysql
/var/lib/mysql/mysql
/usr/include/mysql
/usr/include/mysql/mysql
/usr/bin/mysql
/usr/share/mysql
/usr/lib64/mysql

#4.卸载MySQL各类组件及文件夹
#删除组件
[root@sst_search_a /]# rpm -ev MySQL-devel-5.6.23-1.linux_glibc2.5
[root@sst_search_a /]# rpm -ev MySQL-server-5.6.23-1.linux_glibc2.5
[root@sst_search_a /]# rpm -ev MySQL-client-5.6.23-1.linux_glibc2.5

#删除文件夹
[root@DB-Server init.d]# find / -name mysql
/var/lib/mysql
/var/lib/mysql/mysql
/usr/lib64/mysql
[root@DB-Server init.d]# rm -rf /var/lib/mysql
[root@DB-Server init.d]# rm -rf /var/lib/mysql/mysql
[root@DB-Server init.d]# rm -rf /usr/lib64/mysql

#5.确认MySQL彻底删除
[root@sst_search_a /]# rpm -qa | grep -i mysql

二、 安装MySQL

#当前为root账户
#1.安装server
[root@sst_search_a hive]# rpm -ivh /opt/MySQL-server-5.5.15-1.linux2.6.x86_64.rpm
Preparing...　　　　　　　########################################### [100%]   
1:MySQL-server　　　　　########################################### [100%]   
......  
/usr/bin/mysqladmin -u root password 'new-password'   
/usr/bin/mysqladmin -u root -h test1 password 'new-password'   
......
#查看mysql3306端口
[root@sst_search_a hive]# netstat -nat
tcp　　0　　0 0.0.0.0:3306　　　　 0.0.0.0:*　　　　　 LISTEN

#2.安装client
[root@sst_search_a hive]# rpm -ivh /opt/ MySQL-client-5.5.15-1.linux2.6.x86_64.rpm
warning: MySQL-client-5.1.7-0.i386.rpm: V3 DSA signature: NOKEY, key ID 5072e1f5   
Preparing...　　　　########################################### [100%]   
1:MySQL-client　 ########################################### [100%]

#3.修改密码
[root@sst_search_a hive]# usr/bin/mysqladmin -u root password 'new-password'
#mysqladmin -u用户名 -p旧密码 password 新密码
[root@sst_search_a hive]# mysql -uroot -new-password
mysql> insert into mysql.user(Host,User,Password) values("localhost","hive",password("hive"));
mysql> create database hive;
mysql> grant all on hive.* to hive@'%'  identified by 'hive';
mysql> grant all on hive.* to hive@'localhost'  identified by 'hive';
mysql> flush privileges;
mysql> exit;
[root@sst_search_a hive]# mysql -uhive -phive
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
| test               |
+--------------------+
3 rows in set (0.00 sec)
mysql> exit;

三、安装Hive

#切换至hadoop账户
[hadoop@sst_search_a ~]$ vi  .bash_profile

#HIVE_HOME
export HIVE_HOME=/opt/hive
export PATH=$PATH:$HIVE_HOME/bin

[hadoop@sst_search_a ~]$ source .bash_profile

#配置hive-env.sh
[hadoop@sst_search_a conf]$ cp hive-env.sh.template hive-env.sh
[hadoop@sst_search_a conf]$ vi hive-env.sh

export HIVE_HOME=/opt/hive

[hadoop@sst_search_a conf]$ source hive-env.sh

#配置hive-site.xml
[hadoop@sst_search_a conf]$ cp hive-default.xml.template hive-site.xml
[hadoop@sst_search_a conf]$ vi hive-site.xml
#文件比较大，使用'/javax.jdo.option.ConnectionURL'查找

<property> 
   <name>javax.jdo.option.ConnectionURL </name> 
   <value>jdbc:mysql://localhost:3306/hive </value> 
</property> 
 
<property> 
   <name>javax.jdo.option.ConnectionDriverName </name> 
   <value>com.mysql.jdbc.Driver </value> 
</property>

<property> 
   <name>javax.jdo.option.ConnectionPassword </name> 
   <value>hive </value> 
</property> 
 
<property> 
   <name>hive.hwi.listen.port </name> 
   <value>9999 </value> 
   <description>This is the port the Hive Web Interface will listen on </descript ion> 
</property> 

<property> 
   <name>datanucleus.autoCreateSchema </name> 
   <value>true</value> 
</property> 
 
<property> 
   <name>datanucleus.fixedDatastore </name> 
   <value>false</value> 
</property>

  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/home/hdpsrc/hive/iotmp</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/home/hdpsrc/hive/iotmp</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
    <name>hive.querylog.location</name>
    <value>/home/hdpsrc/hive/iotmp</value>
    <description>Location of Hive run time structured log file</description>
  </property>

#拷贝mysql-connector-java-5.1.6-bin.jar 到hive 的lib下面
[hadoop@sst_search_a conf]$ mv /opt/Desktop/mysql-connector-java-5.1.6-bin.jar /opt/hive/lib/

#把jline-2.12.jar拷贝到hadoop相应的目录下，替代jline-0.9.94.jar，否则启动会报错
[hadoop@sst_search_a conf]$ cp /opt/hive/lib/jline-2.12.jar /opt/hadoop-2.6.0/share/hadoop/yarn/lib/

[hadoop@sst_search_a conf]$ mv /opt/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar /opt/hadoop-2.6.0/share/hadoop/yarn/lib/jline-0.9.94.jar.bak /

#创建hive临时文件夹
[hadoop@sst_search_a conf]$ mkdir /opt/hive/iotmp

#启动hadoop后测试hive
[hadoop@sst_search_a conf]$ hive
hive> show databases;
OK
default
Time taken: 0.907 seconds, Fetched: 1 row(s)


四、遇到的问题

问题1：

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/........../lib/slf4j-log4j12-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/........../slf4j-log4j12-1.5.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

解决办法：
jar包重复，删掉了hive/lib文件夹下的对应jar包

问题2：

Logging initialized using configuration in jar:file:/opt/tuniu/hive/apache-hive-2.1.0-bin/lib/hive-common-2.1.0.jar!/hive-log4j2.properties Async: true
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Permission denied
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:557)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.createTempFile(File.java:2024)
        at org.apache.hadoop.hive.common.FileUtils.createTempFile(FileUtils.java:785)
        at org.apache.hadoop.hive.ql.session.SessionState.createTempFile(SessionState.java:904)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:555)
        ... 9 more

解决办法：
注意到permission denied,一定是权限问题。猜测可能是hadoop用户对hive文件夹权限不够，暴力赋予了足够的权限
chmod 777 -R /opt/hive
（注意这里其实问题变形很多，还是得读一下自己的错误日志）

问题3：

which: no hbase in (/opt/tuniu/java/jdk1.8.0_101/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/tuniu/hadoop/hadoop-2.6.4/bin:/opt/tuniu/hadoop/hadoop-2.6.4/sbin:/root/bin:/opt/tuniu/hive/apache-hive-2.1.0-bin/bin)

还没搞明白和hbase有什么关系。。。不影响使用，暂时搁置。