Mysql、Sqoop、Hive 安装命令总结

最新推荐文章于 2022-11-18 21:15:30 发布

Jmayday

最新推荐文章于 2022-11-18 21:15:30 发布

阅读量194

点赞数 1

分类专栏： Hadoop

本文链接：https://blog.csdn.net/Jmayday/article/details/117366404

版权

Hadoop 专栏收录该内容

19 篇文章 1 订阅

订阅专栏

第一部分 Mysql安装

1、卸载自带的安装包、

rpm -qa | grep -i -E mysql\|mariadb | xargs -n1 sudo rpm -e --nodeps

2、准备好待安装的资源包，上传到/opt/software下

01_mysql-community-common-5.7.29-1.el7.x86_64.rpm
02_mysql-community-libs-5.7.29-1.el7.x86_64.rpm
03_mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm
04_mysql-community-client-5.7.29-1.el7.x86_64.rpm
05_mysql-community-server-5.7.29-1.el7.x86_64.rpm
mysql-connector-java-5.1.48.jar

3、按照顺序安装

sudo rpm -ivh 01_mysql-community-common-5.7.29-1.el7.x86_64.rpm
sudo rpm -ivh 02_mysql-community-libs-5.7.29-1.el7.x86_64.rpm
sudo rpm -ivh 03_mysql-community-libs-compat-5.7.29-1.el7.x86_64.rpm
sudo rpm -ivh 04_mysql-community-client-5.7.29-1.el7.x86_64.rpm
sudo rpm -ivh 05_mysql-community-server-5.7.29-1.el7.x86_64.rpm

4、启动Mysql，并查看密码

sudo systemctl start mysqld
sudo cat /var/log/mysqld.log | grep password

5、配置Mysql

(1)输入密码(错误可对密码加单引号)

mysql -uroot -p’password’

（2）设置密码

mysql> set password=password("Ts73Lzs32");

(3)更改密码策略

mysql> set global validate_password_length=4;
mysql> set global validate_password_policy=0;

（4）设置新密码

mysql> set password=password("000000");

（5）进入mysql

mysql> use mysql

(6)基础操作

1）查询user表
mysql> select user, host from user;
2）修改user表，把Host表内容修改为%
mysql> update user set host="%" where user="root";
3）刷新
mysql> flush privileges;
4）退出
mysql> quit;

第二部分 Sqoop安装

1、下载地址:http://mirrors.hust.edu.cn/apache/sqoop/1.4.6/

2、解压到指定目录

tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /opt/module/

3、修改名称

mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha/ sqoop

4、修改配置文件

(1) 进入到/opt/module/sqoop/conf目录，重命名配置文件

mv sqoop-env-template.sh sqoop-env.sh

(2)编辑内容

vim sqoop-env.sh 

增加如下内容
export HADOOP_COMMON_HOME=/opt/module/hadoop-3.1.3
export HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3
export HIVE_HOME=/opt/module/hive
export ZOOKEEPER_HOME=/opt/module/zookeeper-3.5.7
export ZOOCFGDIR=/opt/module/zookeeper-3.5.7/conf

5、拷贝JD

（1）将mysql-connector-java-5.1.48.jar 上传到/opt/software路径

（2）进入到/opt/software/路径，拷贝jdbc驱动到sqoop的lib目录下。

cp mysql-connector-java-5.1.48.jar /opt/module/sqoop/lib/

6、验证测试

bin/sqoop help

出现以下信息

出现一些Warning警告（警告信息已省略），并伴随着帮助命令的输出：
Available commands:
  codegen            Generate code to interact with database records
  create-hive-table     Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables     Import tables from a database to HDFS
  import-mainframe    Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases        List available databases on a server
  list-tables           List available tables in a database
  merge              Merge results of incremental imports
  metastore           Run a standalone Sqoop metastore
  version            Display version information

测试是否成功连接数据库

bin/sqoop list-databases --connect jdbc:mysql://hadoop102:3306/ --username root --password 000000
出现如下输出：
information_schema
metastore
mysql
oozie
performance_schema

HIve安装

1、解压到指定目录下

tar -zxvf /opt/software/apache-hive-3.1.2-bin.tar.gz -C /opt/module/

2、修改名称

 mv /opt/module/apache-hive-3.1.2-bin/ /opt/module/hive

3、添加环境变量

sudo vim /etc/profile.d/my_env.sh


# 添加内容
#HIVE_HOME
export HIVE_HOME=/opt/module/hive
export PATH=$PATH:$HIVE_HOME/bin

4、使环境变量生效可以重新打开Xshell窗口,或者

 source /etc/profile.d/my_env.sh

5、解决日志Jar包冲突，进入/opt/module/hive/lib目录

mv log4j-slf4j-impl-2.10.0.jar log4j-slf4j-impl-2.10.0.jar.bak

6、HIve元数据配置到mysql

（1）将MySQL的JDBC驱动拷贝到Hive的lib目录下

 cp /opt/software/mysql-connector-java-5.1.48.jar /opt/module/hive/lib/

（2）配置Metastore到MySql

在$HIVE_HOME/conf目录下新建hive-site.xml文件

vim hive-site.xml

添加内容

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://hadoop102:3306/metastore?useSSL=false</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>

    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>000000</value>
    </property>

    <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/user/hive/warehouse</value>
    </property>

    <property>
        <name>hive.metastore.schema.verification</name>
        <value>false</value>
    </property>

    <property>
        <name>hive.metastore.uris</name>
        <value>thrift://hadoop102:9083</value>
    </property>

    <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
    </property>

    <property>
        <name>hive.server2.thrift.bind.host</name>
        <value>hadoop102</value>
    </property>

    <property>
        <name>hive.metastore.event.db.notification.api.auth</name>
        <value>false</value>
    </property>
    
    <property>
        <name>hive.cli.print.header</name>
        <value>true</value>
    </property>

    <property>
        <name>hive.cli.print.current.db</name>
        <value>true</value>
    </property>
</configuration>

7、启动Hive

（1）初始化源数据库

1）登陆MySQL

mysql -uroot -p000000

2）新建Hive元数据库

mysql> create database metastore;

mysql> quit;

3）初始化Hive元数据库

schematool -initSchema -dbType mysql -verbose

（2）启动metastore和hiveserver2

1）Hive 2.x以上版本，要先启动这两个服务，否则会报错：

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

2）在/opt/module/hive/bin目录编写hive服务启动脚本

vim hiveservices.sh

添加内容

#!/bin/bash
HIVE_LOG_DIR=$HIVE_HOME/logs

mkdir -p $HIVE_LOG_DIR

#检查进程是否运行正常，参数1为进程名，参数2为进程端口
function check_process()
{
    pid=$(ps -ef 2>/dev/null | grep -v grep | grep -i $1 | awk '{print $2}')
    ppid=$(netstat -nltp 2>/dev/null | grep $2 | awk '{print $7}' | cut -d '/' -f 1)
    echo $pid
    [[ "$pid" =~ "$ppid" ]] && [ "$ppid" ] && return 0 || return 1
}

function hive_start()
{
    metapid=$(check_process HiveMetastore 9083)
    cmd="nohup hive --service metastore >$HIVE_LOG_DIR/metastore.log 2>&1 &"
    cmd=$cmd" sleep 4; hdfs dfsadmin -safemode wait >/dev/null 2>&1"
    [ -z "$metapid" ] && eval $cmd || echo "Metastroe服务已启动"
    server2pid=$(check_process HiveServer2 10000)
    cmd="nohup hive --service hiveserver2 >$HIVE_LOG_DIR/hiveServer2.log 2>&1 &"
    [ -z "$server2pid" ] && eval $cmd || echo "HiveServer2服务已启动"
}

function hive_stop()
{
    metapid=$(check_process HiveMetastore 9083)
    [ "$metapid" ] && kill $metapid || echo "Metastore服务未启动"
    server2pid=$(check_process HiveServer2 10000)
    [ "$server2pid" ] && kill $server2pid || echo "HiveServer2服务未启动"
}

case $1 in
"start")
    hive_start
    ;;
"stop")
    hive_stop
    ;;
"restart")
    hive_stop
    sleep 2
    hive_start
    ;;
"status")
    check_process HiveMetastore 9083 >/dev/null && echo "Metastore服务运行正常" || echo "Metastore服务运行异常"
    check_process HiveServer2 10000 >/dev/null && echo "HiveServer2服务运行正常" || echo "HiveServer2服务运行异常"
    ;;
*)
    echo Invalid Args!
    echo 'Usage: '$(basename $0)' start|stop|restart|status'
    ;;
esac

8、基本操作

#添加执行权限
 chmod +x hiveservices.sh

#启动Hive后台服务
hiveservices.sh start

#查看Hive后台服务运行情况
 hiveservices.sh status
Metastore服务运行正常
HiveServer2服务运行异常

#启动Hive客户端
 bin/hive