环境
CentOS 7
hadoop 2.7.3 CentOS 7.0安装flume
hive 2.1.1
准备
下载hive
hive官方下载地址: http://www.apache.org/dyn/closer.cgi/hive/
然后上传到namenode机器上
说明: 由于我用不了wget,能用wget的可以选择以下方法下载hive-2.1.1
wget http://mirrors.ustc.edu.cn/apache/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz
解压
找到hive的压缩包(我放在了/data/software 目录下),解压
cd /data/software
ls
mkdir /opt/hive
tar -zxvf apache-hive-2.1.1-bin.tar.gz -C /opt/hive/
cd /opt/hive
mv apache-hive-2.1.1-bin apache-hive-2.1.1
安装
配置环境变量
vi /etc/profile
添加上hive的相关配置
export HIVE_HOME=/opt/hive/apache-hive-2.1.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin
使其生效
source /etc/profile
配置hive
拷贝出hive的模板,并编辑查看
cd /opt/hive/apache-hive-2.1.1/conf
ls
cp hive-default.xml.template hive-site.xml
vi hive-site.xml
观察到配置文件里有
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
<description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.</description>
</property>
所以需要在hadoop集群中创建相应目录(需要先启动hadoop)。
hadoop fs -mkdir -p /user/hive/warehouse
hadoop fs -mkdir -p /tmp/hive
hadoop fs -chmod -R 777 /user/hive/warehouse
hadoop fs -chmod -R 777 /tmp/hive
hadoop fs -ls /user/hive
hadoop fs -ls /tmp
编辑hive-site.xml
cd /opt/hive/apache-hive-2.1.1/conf
vi hive-site.xml
修改hive-site.xml文件配置,由于我linux命令不是太熟悉,所以用的方法是将文件拷贝到本地,用EditPlus编辑器做的文件修改,然后上传到机器上。
修改临时目录和用户
cd /opt/hive/apache-hive-2.1.1 mkdir tmp chmod 777 tmp
将tmp目录的路径配置到hive-site.xml中,同时将用户替换为root (将${system:java.io.tmpdir}全都替换为tmp目录,将${system:user.name}全都替换为root)
<property> <name>hive.exec.local.scratchdir</name> <!-- value>${system:java.io.tmpdir}/${system:user.name}</value --> <value>/opt/hive/apache-hive-2.1.1/tmp/root</value> <description>Local scratch space for Hive jobs</description> </property> <property> <name>hive.downloaded.resources.dir</name> <!--value>${system:java.io.tmpdir}/${hive.session.id}_resources</value --> <value>/opt/hive/apache-hive-2.1.1/tmp/${hive.session.id}_resources</value> <description>Temporary local directory for added resources in the remote file system.</description> </property>
修改mysql数据库相关配置
将javax.jdo.option.ConnectionDriverName 的value换为mysql的驱动路径com.mysql.jdbc.Driver
<property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property>
将javax.jdo.option.ConnectionURL 的value换为mysql的地址
<property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://192.168.122.128:3306/hive?createDatabaseIfNotExist=true</value> <description> JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. </description> </property>
将javax.jdo.option.ConnectionUserName 的value换为mysql数据库的登录名
<property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>Username to use against metastore database</description> </property>
将javax.jdo.option.ConnectionPassword 的value换为mysql数据库的登录密码
<property> <name>javax.jdo.option.ConnectionPassword</name> <value>你的密码</value> <description>password to use against metastore database</description> </property>
将mysql数据库驱动上传到hive的lib目录下
mysql驱动下载地址(mysql-connector-java-5.1.40-bin.jar):https://pan.baidu.com/s/1slsQBpj 密码:vxqd
修改hive-env.sh
cp hive-env.sh.template hive-env.sh vi hive-env.sh
添加上hadoop和hive的环境变量
export HADOOP_HOME=/opt/hadoop/hadoop-2.7.3 export HIVE_CONF_DIR=/opt/hive/apache-hive-2.1.1/conf export HIVE_AUX_JARS_PATH=/opt/hive/apache-hive-2.1.1/lib
source hive-env.sh
启动测试
启动mysql
cd /opt/hive/apache-hive-2.1.1/bin
schematool -initSchema -dbType mysql
通过mysql客户端能够查看到hive数据库的相关信息;
我是通过pycharm成功连接mysql数据库并能够看到hive数据库的一些信息
select t.* from TBLS t;
启动hive
./hive
一切顺利,没有报错
测试hive
show functions;
并且能够执行数据库的相关操作,如建库,建表等
参考: Hive教程
总结
此次安装hive过程很顺利,基本上参考了一篇blog就能够顺利完成,感谢dalao的博客。
参考:CentOS7基于Hadoop 2.7.3安装Hive 2.1.1
Hive教程