是什么
Hive是Hadoop生态的数据仓库工具 Hive将存储在HDFS上的文件映射为关系表 通过解析开发者提交的SQL语句,将SQL转换成MR任务,提交到Hadoop执行 Hive提供了命令行客户端和JDBC
Hive架构
Hive 安装
将Hive的元数据库替换为mysql
-
hive自带关系型数据库derby用于存储hive中的库和表的结构信息(metadata)
-
生产环境中通常将derby替换为mysql等性能更好的开源数据库
1、yum
yum install mysql
2、使用安装包进行离线安装
3、上传安装包
mysql-8.0.25-linux-glibc2.12-x86_64.tar.xz安装包到Linux集群的 /opt/
-
下载mysql安装包
-
使用脚本自动化安装 ,创建 mysql8installer.sh
#!/bin/bash echo -e "\033[4;40;31m欢迎使用mysql离线安装自动化脚本 v1.0\033[0m" echo -e "\033[4;40;31m作者:Amos QQ:410507803 E-Mail:amos@amoscloud.com\033[0m\n" read -p "请输入mysql8的zx压缩包文件所在路径(eg:/opt/mysql8.xxx.xz):" FILE_PATH read -p "请输入想要安装的目录(eg:/usr/local/mysql):" DEST_PATH rpm -e --nodeps $(rpm -qa | grep mariadb) echo -e "\033[40;32m (1/13)正在解压,请耐心等待解压过程约1-3分钟... \033[0m" tar Jxf $FILE_PATH -C . echo -e "\033[40;32m 解压完成 \033[0m" echo -e "\033[40;32m (2/13)移动加压后的文件到$DEST_PATH \033[0m" mv mysql-8*x86_64 $DEST_PATH echo -e "\033[40;32m (3/13)添加环境变量$DEST_PATH \033[0m" echo "export MYSQL_HOME=$DEST_PATH" >>/etc/profile echo 'export PATH=.:$MYSQL_HOME/bin:$PATH' >>/etc/profile source /etc/profile echo -e "\033[40;32m (4/13)创建data目录 \033[0m" mkdir $DEST_PATH/data echo -e "\033[40;32m (5/13)创建my.cnf配置文件 \033[0m" rm -rf /etc/my.cnf echo " [client] port=3306 socket=/tmp/mysql.sock [mysqld] port=3306 user=mysql socket=/tmp/mysql.sock basedir=$DEST_PATH datadir=$DEST_PATH/data log-error=$DEST_PATH/error.log pid-file = $DEST_PATH/mysql.pid transaction_isolation = READ-COMMITTED character-set-server = utf8 collation-server = utf8_general_ci lower_case_table_names = 1 " > /etc/my.cnf echo 'sql_mode = "STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO"' >> /etc/my.cnf echo -e "\033[40;32m (6/13)创建mysql组 \033[0m" groupadd mysql echo -e "\033[40;32m (7/13)创建mysql用户并加入mysql组 \033[0m" useradd -g mysql mysql echo -e "\033[40;32m (8/13)修改安装目录权限和所有者 \033[0m" chown -R mysql:mysql $DEST_PATH chmod -R 755 $DEST_PATH echo -e "\033[40;32m (9/13)初始化mysql \033[0m" $DEST_PATH/bin/mysqld --initialize --user=mysql echo -e "\033[40;32m (10/13)尝试启动mysql \033[0m" $DEST_PATH/support-files/mysql.server start echo -e "\033[40;32m (11/13)将mysqld添加为服务并设置开机自启动 \033[0m" cp $DEST_PATH/support-files/mysql.server /etc/init.d/mysqld chmod 755 /etc/init.d/mysqld chkconfig --add mysqld chkconfig --level 345 mysqld on echo -e "\033[40;32m (12/13)重启mysql \033[0m" service mysqld restart echo -e "\033[40;32m (13/13)读取临时密码 \033[0m" TEMP_PW=$(cat $DEST_PATH/error.log | grep 'password' | awk -F' ' '{print $NF}') echo -e " \033[40;32m mysql的初始临时密码为:$TEMP_PW \033[0m \033[40;32m 使用初始密码登录mysql后,您可以使用如下SQL修改初始密码: \033[0m \033[40;33m ALTER user 'root'@'localhost' IDENTIFIED BY 'a123456'; \033[0m \033[40;32m 使用如下SQL添加可远程访问的root用户: \033[0m \033[40;33m CREATE USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'a123456'; \033[0m \033[40;33m GRANT ALL ON *.* TO 'root'@'%'; \033[0m \033[40;33m FLUSH PRIVILEGES; \033[0m \033[40;32m 3秒后将使用初始密码登录mysql,感谢您的使用 \033[0m " sleep 3 mysql -uroot -p$TEMP_PW
4、根据提示填写路径
安装路径:/usr/local/
5、成功登录mysql,修改密码
本机登录
userName:root
passWord:123456
mysql> ALTER user 'root'@'localhost' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.01 sec) mysql> CREATE USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY '123456'; Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL ON *.* TO 'root'@'%'; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec)
6、退出mysql
mysql> exit Bye
7、为Hive创建数据仓库
在idea等数据库管理工具,连接到linux的mysql,创建数据仓库
create schema metastore character set 'utf8';