一、部署流程
安装自动化工具——环境配置——部署OCP——部署集群——部署OBProxy
二、部署前准备
在最小化部署时,一般需要3台OBSERVER服务器安装OceanBase集群和一台单独的服务器安装OCP用于管理OceanBase集群。
OceanBase数据库集群至少由三个节点组成,每个节点对应一个observer进程,不同节点上的多个observer进程组成一个集群对外提供服务;
- 服务器的规划及使用
observer:10.2.12.50-52
obproxy:10.2.12.50
ocp:10.2.12.49
- 安装包
metaob-OB2277_OBP320_x86_20220429.tar.gz //用于部署OCP MetaDB
oceanbase-3.2.4.4-104030012023070623.el7.x86_64.rpm //用于部署OB集群
ocp315.tar.gz //用于部署OCP
t-oceanbase-antman-1.4.2-20220511175709.alios7.x86_64.rpm //用于部署OAT
三、开始安装
为更能了解OB集群,本次采用命令行部署;
1、部署环境配置
- 安装oat_cli
oat_cli是OB数据库生态产品的自动化部署工具;
它提供一键部署OCP、OMS、ODC等OB周边工具平台的cli工具;
root:(所有服务器都要执行)
rpm -ivh t-oceanbase-antman-1.4.2-20220511175709.alios7.x86_64.rpm
2、通过oat_cli配置部署环境
- 为每台服务器添加admin用户。
cd /root/t-oceanbase-antman/clonescripts
./clone.sh -u
[root@baseline clonescripts]# id admin
uid=500(admin) gid=500(admin) groups=500(admin)
-
修改操作系统内核配置
OCP服务器
cd /root/t-oceanbase-antman/clonescripts
./clone.sh -c -r ocp -
OBServer服务器
cd /root/t-oceanbase-antman/clonescripts
./clone.sh -c -r ob -
安装依赖包
OCP服务器
cd /root/t-oceanbase-antman/clonescripts
./clone.sh -m -r ocp -
OBServer服务器
cd /root/t-oceanbase-antman/clonescripts
./clone.sh -m -r ob -
安装Docker
在OCP与OMS服务器上执行
cd /root/t-oceanbase-antman/clonescripts
./clone.sh -i
3、配置时间同步
OceanBase产品的所有服务器的物理时间要保持同步,误差尽量控制在100ms以内。
因为时钟配置同步服务NTP或者chrony。
以下采用NTP的方法:
安装NTP:
yum install ntp ntpdate -y
配置ntp.conf文件
vi /etc/ntp.conf
server 127.127.1.0
重启NTP服务
systemctl restart ntpd
检查ntp状态
ntpstat
ntpq -p
检查时延
clockdiff 196.168.x.x
手工同步
ntpdate -u 196.168.x.x
4、部署前环境检查
如果有显示FAIL的信息,根据报错提示进行修改即可
-
OCP服务器:
cd /root/t-oceanbase-antman/clonescripts/
sh precheck.sh -m ocp -
OBServer服务器:
cd /root/t-oceanbase-antman/clonescripts/
sh precheck.sh -m ob -
oms服务器:
cd /root/t-oceanbase-antman/clonescripts/
sh precheck.sh -m oms
5、部署OCP
OCP可以单节点部署,也可以多节点。本次采用单节点部署:
- 以root用户登录OCP服务器,执行以下命令
cd /root/t-oceanbase-antman
bash init_obcluster_conf.sh
[root@baseline t-oceanbase-antman]# bash init_obcluster_conf.sh
/root/t-oceanbase-antman/obcluster.conf NOT exists, try to generate /root/t-oceanbase-antman/obcluster.conf
install ocp with 1 machine or 3 machines? [1/3] 1
根据注释修改obcluster.conf
(填写机器IP和 metaob 容器内的root/admin密码,LB_MODE=none)
## obcluster.conf
##
SINGLE_OCP_MODE=TRUE
################################ 根据环境必须修改 / MUST CHANGE ACCORDING ENVIRONMENT ################################
############ 填写机器IP和 metaob 容器内的root/admin密码 / Edit Machine IP and Password Of root/admin ############
ZONE1_RS_IP=10.2.12.49
OBSERVER01_ROOTPASS='' # metaob容器使用,非宿主机账号,宿主机SSH信息请使用指定环境变量的方式,详见 install.sh -h
OBSERVER01_ADMINPASS=''
############ 填写负载均衡配置 / Edit Configuration of Load Balance ############
LB_MODE=none
############ 根据服务器CPU、内存设置容器资源编排 / Allocate Container Resources According To Server ############
OB_DOCKER_CPUS=24
OB_DOCKER_MEMORY=64G
OB_SYSTEM_MEMORY=20G # Do not modify OS_SYSTEM_MEMORY unless you know what you are doing!
OCP_DOCKER_CPUS=4
OCP_DOCKER_MEMORY=8G
OBPROXY_DOCKER_CPUS=2
OBPROXY_DOCKER_MEMORY=10G
-
开始安装
cd /root/t-oceanbase-antman
./install.sh -h
export SSH_AUTH=password
export SSH_USER=root
export SSH_PORT=22
export SSH_PASSWORD=‘bbBB11__’
export SSH_KEY_FILE=/root/.ssh/id_rsa
bash install.sh -i 1-8 //安装步骤1~8 -
部署后检查:
浏览器登录OCP
http://<ocp服务器IP>:8080
ocp默认密码:aaAA11__或root
6、部署三副本OB集群
OceanBase集群三副本,至少要有三台主机,分为三个zone。
OB是单进程软件,只需要安装oceanbase-xxx.rpm包和obproxy-xxxx.rpm包;
-
各主机分别安装数据库rpm包
cd $rpm_dir
rpm -ivh oceanbase-3.2.4.4-104030012023070623.el7.x86_64.rpm -
清理旧OB目录(第一次部署不用)
su - admin
kill -9pidof observer
rm -rf /data/1/obdemo
rm -rf /data/log1/obdemo
rm -rf /home/admin/oceanbase/store/obdemo /home/admin/oceanbase/log/* /home/admin/oceanbase/etc/config
ps -ef | grep observer -
初始化目录
mkdir -p /data/1/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/1/obdemo/ t / h o m e / a d m i n / o c e a n b a s e / s t o r e / o b d e m o / t /home/admin/oceanbase/store/obdemo/ t/home/admin/oceanbase/store/obdemo/t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/obdemo/ t / h o m e / a d m i n / o c e a n b a s e / s t o r e / o b d e m o / t /home/admin/oceanbase/store/obdemo/ t/home/admin/oceanbase/store/obdemo/t; done
obdemo:是以集群名称创建的目录,可自定义,启动进程时会用到;
-
检查结果
cd /home/admin/oceanbase
tree store* [root@baseline oceanbase]# tree store store └── obdemo ├── clog -> /data/log1/obdemo/clog ├── etc2 -> /data/log1/obdemo/etc2 ├── etc3 -> /data/1/obdemo/etc3 ├── ilog -> /data/log1/obdemo/ilog ├── oob_clog -> /data/log1/obdemo/oob_clog ├── slog -> /data/log1/obdemo/slog ├── sort_dir -> /data/1/obdemo/sort_dir └── sstable -> /data/1/obdemo/sstable 9 directories, 0 files
-
安装OceanBase客户端
OBClinet是Oceanbase命令行客户端,可以访问OceanBase的MYSQL租户和ORACLE租户
rpm -ivh obclient-2.2.3-20230808103716.el7.x86_64.rpm
[root@baseline install]# which obclient
/bin/obclient
7、初始化OceanBase集群
- 启动节点observer进程
在每个节点的admin用户下,启动observer进程。
节点一:
su - admin
cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i ens192 -P2882 -p2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r ‘10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881’ -c 10001 -n obdemo -o “system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2”
[admin@baseline obdemo]$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r '10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
/home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/store/obdemo -r 10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
devname: ens192
rpc port: 2882
mysql port: 2881
zone: zone1
data_dir: /home/admin/oceanbase/store/obdemo
rs list: 10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881
cluster id: 10001
appname: obdemo
optstr: system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
节点二:
su - admin
cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r ‘10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881’ -c 10001 -n obdemo -o “system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2”
[admin@baseline ~]$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r '10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
/home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone2 -d /home/admin/oceanbase/store/obdemo -r 10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
devname: ens192
rpc port: 2882
mysql port: 2881
zone: zone2
data_dir: /home/admin/oceanbase/store/obdemo
rs list: 10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881
cluster id: 10001
appname: obdemo
optstr: system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
节点三:
su - admin
cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone3 -d /home/admin/oceanbase/store/obdemo -r ‘10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881’ -c 10001 -n obdemo -o “system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2”
[admin@baseline oceanbase]$ cd /home/admin/oceanbase && /home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone3 -d /home/admin/oceanbase/store/obdemo -r '10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881' -c 10001 -n obdemo -o "system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2"
/home/admin/oceanbase/bin/observer -i ens192 -P 2882 -p 2881 -z zone3 -d /home/admin/oceanbase/store/obdemo -r 10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881 -c 10001 -n obdemo -o system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
devname: ens192
rpc port: 2882
mysql port: 2881
zone: zone3
data_dir: /home/admin/oceanbase/store/obdemo
rs list: 10.2.12.50:2882:2881;10.2.12.51:2882:2881;10.2.12.52:2882:2881
cluster id: 10001
appname: obdemo
optstr: system_memory=30G,datafile_size=90G,config_additional_dir=/data/1/obdemo/etc3;/data/log1/obdemo/etc2
可通过netstat -nltp命令查看observer进程是否启动成功。如果监听到2881与2882端口,则说明启动成功。
[admin@baseline oceanbase]$ netstat -nltp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:2881 0.0.0.0:* LISTEN 78959/observer
tcp 0 0 0.0.0.0:2882 0.0.0.0:* LISTEN 78959/observer
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:3939 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 ::1:631 :::* LISTEN -
tcp6 0 0 :::2881 :::* LISTEN 78959/observer
tcp6 0 0 :::10050 :::* LISTEN -
tcp6 0 0 :::2882 :::* LISTEN 78959/observer
- 集群bootstrap操作
通过obclient命令连接任意一台节点,密码为空
obclient -h10.2.12.50 -uroot -P2881 -p
set session ob_query_timeout=1000000000;
alter system bootstrap ZONE ‘zone1’ SERVER ‘10.2.12.50:2882’,ZONE ‘zone2’ SERVER ‘10.2.12.51:2882’,ZONE ‘zone3’ SERVER ‘10.2.12.52:2882’;
注:这一步失败报错了,其原因很可能就是三节点observer进程启动参数有不对,目录权限不对,日志目录空间不足、时间不同步、节点内存资源不足;
从头开始请清理OceanBase目录
- 验证集群初始化成功
上面命令正常执行后,可看到oceanbase库
show databases;
obclient [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| oceanbase |
| information_schema |
| mysql |
| SYS |
| LBACSYS |
| ORAAUDITOR |
| test |
+--------------------+
- 修改密码
sys租户的root用户密码默认为空,初始化成功后请修改密码。
alter user root identified by “bbBB22__”;
8、安装OBProxy
- 安装OBProxy RPM包
su - root
rpm -ivh obproxy-1.8.6-20210210153306.el7.x86_64.rpm
注:obproxy安装目录在/opt/taobao/install
-
创建obproxy的软链接
cd /opt/taobao/install
ln -s obproxy-1.8.6 obproxy -
建立obproxy进程运行日志目录
su - admin
mkdir -p /home/admin/logs/obproxy/log
for t in {log};do ln -s /opt/taobao/install/obproxy/ t / h o m e / a d m i n / l o g s / o b p r o x y / l o g / t /home/admin/logs/obproxy/log/ t/home/admin/logs/obproxy/log/t; done
tree /opt/taobao/install/obproxy -
初始化OBProxy
OBProxy需要跟后端OBServer节点保持通信。所以需要提前在OB集群sys租户下为OBProxy创建连接用户(proxyro)和密码。
一个OceanBase集群对应一个proxyro账号。
创建proxyro用户并授予select权限
obclient -h10.2.12.50 -uroot@sys -P2881 -p -c -A //登录sys租户
create user proxyro identified by “”;
grant select on . to proxyro;
show grants for proxyro; -
启动OBProxy
在admin用户下并在OBProxy软件的home目录下执行;
su - admin
cd /opt/taobao/install/obproxy && bin/obproxy -r “10.2.12.50:2881;10.2.12.51:2881;10.2.12.52:2881” -p 2883 -o “skip_proxy_sys_private_check=true,enable_strict_kernel_release=false,enable_cluster_checkout=false,enable_metadb_used=false” -c obdemo -
查看进程是否存在
ps -ef |grep obproxy -
停止OBProxy的方法是直接kill
kill -9pidof obproxy
再次启动OBProxy不需要指定参数。参数已写到参数文件里
cd /opt/taobao/install/obproxy && bin/obproxy -
修改obproxy参数
obclient -h10.2.12.50 -uroot@proxysys -P2883 -p
这里租户的名称是:proxysys。默认密码为空
obclient [(none)]> show proxyconfig like '%sys_password%';
+------------------------+-------+--------------------------------+-------------+---------------+
| name | value | info | need_reboot | visible_level |
+------------------------+-------+--------------------------------+-------------+---------------+
| observer_sys_password1 | | password for observer sys user | false | SYS |
| observer_sys_password | | password for observer sys user | false | SYS |
| obproxy_sys_password | | password for obproxy sys user | false | SYS |
+------------------------+-------+--------------------------------+-------------+---------------+
alter proxyconfig set observer_sys_password='123456';
alter proxyconfig set obproxy_sys_password='123456';
alter proxyconfig set observer_sys_password=‘123456’;
alter proxyconfig set obproxy_sys_password=‘123456’;
- 连接测试
obclient -h10.2.12.50 -P2883 -uroot@sys#obdemo -p -c -A oceanbase
obclient [oceanbase]> show full processlist;
+------------+---------+--------+------------------+-----------+---------+------+--------+-----------------------+------------+------+--------------------+
| Id | User | Tenant | Host | db | Command | Time | State | Info | Ip | Port | Proxy_sessid |
+------------+---------+--------+------------------+-----------+---------+------+--------+-----------------------+------------+------+--------------------+
| 3221598964 | root | sys | 10.2.12.50:63882 | oceanbase | Query | 0 | ACTIVE | show full processlist | 10.2.12.50 | 2881 | 721152299409539074 |
| 3221749976 | proxyro | sys | 10.2.12.50:45966 | oceanbase | Sleep | 12 | SLEEP | NULL | 10.2.12.51 | 2881 | 721152299409539075 |
+------------+---------+--------+------------------+-----------+---------+------+--------+-----------------------+------------+------+--------------------+
obclient [oceanbase]> select * from __all_server;
+----------------------------+----------------------------+------------+----------+----+-------+------------+-----------------+--------+-----------------------+-------------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+------------+----------+----+-------+------------+-----------------+--------+-----------------------+-------------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2023-11-16 16:39:58.509119 | 2023-11-16 16:40:18.642282 | 10.2.12.50 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.2.4.4_104030012023070623-d20b26f9031acf5186a69b90fd834db235bd7ef5(Jul 6 2023 23:12:44) | 0 | 1700124017658114 | 0 | 1 | 0 |
| 2023-11-16 16:39:58.503785 | 2023-11-16 16:40:20.444545 | 10.2.12.51 | 2882 | 2 | zone2 | 2881 | 0 | active | 0 | 3.2.4.4_104030012023070623-d20b26f9031acf5186a69b90fd834db235bd7ef5(Jul 6 2023 23:12:44) | 0 | 1700124017460032 | 0 | 1 | 0 |
| 2023-11-16 16:39:57.504540 | 2023-11-16 16:40:21.993712 | 10.2.12.52 | 2882 | 3 | zone3 | 2881 | 0 | active | 0 | 3.2.4.4_104030012023070623-d20b26f9031acf5186a69b90fd834db235bd7ef5(Jul 6 2023 23:12:44) | 0 | 1700124017755337 | 0 | 1 | 0 |
+----------------------------+----------------------------+------------+----------+----+-------+------------+-----------------+--------+-----------------------+-------------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
3 rows in set (0.026 sec)
注:obproxy启动后,还需要修改proxyro的密码(通过proxy参数obproxy_sys_password)指定,设置为跟OB集群里创建的proxyro密码一致才能链接那个OB集群