文章目录
Hive管理记录
一、Linux下启动hive服务(beeline)
1.启动metastore
[root@hadoop bin]# hive --service metastore &
[1] 3638
2.启动hiveserver2
[root@hadoop bin]# hive --service hiveserver2 &
[2] 3718
3.beeline连接数据库
[root@hadoop bin]# beeline
Beeline version 1.2.1 by Apache Hive
beeline> !connect jdbc:hive2://hadoop:10000
Connecting to jdbc:hive2://hadoop:10000
Enter username for jdbc:hive2://hadoop:10000: root
Enter password for jdbc:hive2://hadoop:10000: ******
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hadoop:10000>
二、Supervisor管理hive服务(metastore,hiveserver2),防止意外杀死Hive服务，导致任务中断
在做通过hudi采集数据并落地到hive的过程中，因为经常有Hive的服务被杀死，所以想到用supervisor来管理hive的进程，当hive的服务被意外杀死后，实现自动启动。
使用supervisor管理进程
1. 安装supervisor
1.1 安装
##1. 安装
yum -y install epel-release
yum -y install supervisor
##2. 配置
mv /etc/supervisord.conf /etc/supervisord.conf.bak
1.2 配置
## vi /etc/supervisord.conf
; filename:supervisord.conf
; author:zxy
; date:2021-07-18
[unix_http_server]
file=/var/run/supervisor/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; sockef file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
[inet_http_server] ; inet (TCP) server disabled by default
port=0.0.0.0:9001 ; (ip_address:port specifier, *:port for all iface)
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=1024 ; (min. avail process descriptors;default 200)
user=root
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL for a unix socket
serverurl=http://0.0.0.0:9001 ; use an http:// url to specify an inet socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
[include]
files = /etc/supervisord.d/*.conf
1.3 启动supervisord
[root@hadoop /]# touch /var/run/supervisor/supervisord.pid
/etc/supervisord.d/ : 存放supervisor的配置文件
/var/run/supervisor/: 存放pid的进程
/var/log/supervisor/: 存放supervisor的日志文件
##2. 设置开机自启动
[root@hadoop supervisor]# systemctl enable supervisord
##3. 开启服务器
[root@hadoop supervisor]# systemctl start supervisord
[root@hadoop supervisor]# systemctl status supervisord
● supervisord.service - Process Monitoring and Control Daemon
Loaded: loaded (/usr/lib/systemd/system/supervisord.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2021-05-27 14:50:01 CST; 12s ago
Process: 16460 ExecStart=/usr/bin/supervisord -c /etc/supervisord.conf (code=exited, status=0/SUCCESS)
Main PID: 16481 (supervisord)
CGroup: /system.slice/supervisord.service
└─16481 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
May 27 14:50:01 hadoop systemd[1]: Starting Process Monitoring and Control Daemon...
May 27 14:50:01 hadoop systemd[1]: Started Process Monitoring and Control Daemon.
1.4 脚本管理supervisord启动
#!/bin/bash
# filename:start-supervisord-hive.sh
# autho:zxy
# date:2021-07-19
# 接受参数
CMD=$1
## 帮助函数
usage() {
echo "usage:"
echo "start-supervisord-hive.sh start/stop"
echo "description:"
echo " start:start supervisord-hive"
echo " stop:stop supervisord-hive"
exit 0
}
if [ ${CMD} == "start" ];then
# 启动supervisor管理的hive服务
systemctl start supervisord
systemctl status supervisord
elif [ ${CMD} == "stop" ];then
# 关闭supervisor管理的hive服务
systemctl stop supervisord
systemctl status supervisord
else
usage
fi
2. supervisor管理进程
2.1 hive_metastore.conf
[root@hadoop supervisord.d]# vim hive_metastore.conf
; filename:hive_metastore.conf
; author:zxy
; date:2021-07-18
; desc:配置supervisor管理metastore
[program:metastore]
directory=/opt/apps/hive-1.2.1 ;软件的前缀
command=/opt/apps/hive-1.2.1/bin/hive --service metastore & ; 配置启动metastore的命令
stderr_logfile=/var/log/supervisor/hivemetastore.err ; 错误日志存放路径
stdout_logfile=/var/log/supervisor/hivemetastore.log ; 标准的日志存放路径
stdout_logfile_maxbytes=10MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
user=root ; 只能root用户启动
autostart=true ; 自动启动
autorestart=true ; 自动重启
startsecs=10 ; 开机10s之后如果应用程序是running状态，supervisor就认为这个启动是成功
startretries=3 ; 重启3此之后还是没有成功，就标记为失败
redirect_stderr=false ; 如果是true，那么stderr日志就会写入到stdout中
2.2 hive_hiveserver2.conf
[root@hadoop supervisord.d]# vim hive_hiveserver2.conf
; filename:hive_hiveserver2.conf
; author:zxy
; date:2021-07-18
; desc:配置supervisor管理hiveserver2
[program:hiveserver]
directory=/opt/apps/hive-1.2.1 ;软件的前缀
command=/opt/apps/hive-1.2.1/bin/hive --service hiveserver2 & ; 配置启动hiveserver的命令
stderr_logfile=/var/log/supervisor/hiveserver.err ; 错误日志存放路径
stdout_logfile=/var/log/supervisor/hiveserver.log ; 标准的日志存放路径
stdout_logfile_maxbytes=10MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
user=root ; 只能root用户启动
autostart=true ; 自动启动
autorestart=true ; 自动重启
startsecs=10 ; 开机10s之后如果应用程序是running状态，supervisor就认为这个启动是成功
startretries=3 ; 重启3此之后还是没有成功，就标记为失败
redirect_stderr=false ; 如果是true，那么stderr日志就会写入到stdout中
3.通过supervisor开启/关闭/重启hive服务
##1. 查看supervisor下的所有的应用程序的状态
[root@hadoop azkaban-solo-server]# supervisorctl reread
metastore: available
hiveserver: available
##2. 更新supervisor中的指定程序
[root@hadoop azkaban-solo-server]# supervisorctl update metastore
metastore: added process group
[root@hadoop azkaban-solo-server]# supervisorctl update hiveserver
hiveserver: added process group
##3. 查看指定程序的状态
[root@hadoop azkaban-solo-server]# supervisorctl status metastore
metastore RUNNING pid 25360, uptime 0:01:05
...
##4. 关闭指定的程序
[root@hadoop azkaban-solo-server]# supervisorctl stop metastore
metastore: stopped
...
##5. 开启指定的程序
[root@hadoop azkaban-solo-server]# supervisorctl start metastore
metastore: stopped
...
4. supervisorctl 启动hive的进程
[root@hadoop supervisord.d]# supervisorctl start metastore
[root@hadoop supervisord.d]# supervisorctl start hiveserver
[root@hadoop supervisord.d]# jps
3361 NodeManager
2456 NameNode
2601 DataNode
48601 RunJar
3242 ResourceManager
48602 RunJar
2845 SecondaryNameNode
48831 Jps
[root@hadoop supervisord.d]# hive
Logging initialized using configuration in file:/data/apps/hive-1.2.1/conf/hive-log4j.properties
hive (default)>
三、set hive.fetch.task.conversion=more；设置Fetch 抓取
Hive 中对某些情况的查询可以不必使用 MapReduce 计算。
例如： SELECT * FROM employees;
在这种情况下，Hive 可以简单地读取 employee 对应的存储目录下的文件，然后输出查询结果到控制台。
1）把 hive.fetch.task.conversion 设置成 none，然后执行查询语句，都会执行 mapreduce
程序。
hive (zxy)> set hive.fetch.task.conversion=none;
2）把 hive.fetch.task.conversion 设置成 more， 然后执行查询语句， 如下查询方式都不
会执行 mapreduce 程序。
hive (zxy)> set hive.fetch.task.conversion=more;
扫一扫
5682
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
