实践练习二(必选):手动部署 OceanBase 3.1.2集群(离线安装)

手动部署 OceanBase 集群

环境准备和检查

​ 申请一台物理机,作为部署使用,然后对其进行环境参数检查

操作系统检查

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@localhost ~]# 

内存

[root@localhost ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           251G         15G        215G         86M         19G        234G
Swap:           15G          0B         15G

CPU

  [root@localhost ~]# cat /proc/cpuinfo | grep processor
  processor       : 0
  processor       : 1
  processor       : 2
  processor       : 3
  processor       : 4
  processor       : 5
  processor       : 6
  processor       : 7
  processor       : 8
  processor       : 9
  processor       : 10
  processor       : 11
  processor       : 12
  processor       : 13
  processor       : 14
  processor       : 15
  processor       : 16
  processor       : 17
  processor       : 18
  processor       : 19
  processor       : 20
  processor       : 21
  processor       : 22
  processor       : 23
  processor       : 24
  processor       : 25
  processor       : 26
  processor       : 27
  processor       : 28
  processor       : 29
  processor       : 30
  processor       : 31
  processor       : 32
  processor       : 33
  processor       : 34
  processor       : 35
  processor       : 36
  processor       : 37
  processor       : 38
  processor       : 39
  processor       : 40
  processor       : 41
  processor       : 42
  processor       : 43
  processor       : 44
  processor       : 45
  processor       : 46
  processor       : 47
  processor       : 48
  processor       : 49
  processor       : 50
  processor       : 51
  processor       : 52
  processor       : 53
  processor       : 54
  processor       : 55
  processor       : 56
  processor       : 57
  processor       : 58
  processor       : 59
  processor       : 60
  processor       : 61
  processor       : 62
  processor       : 63

配置yum源

下载软件源配置文件,刷新缓存。

由于我的机器是内网环境,并且有内网固定源地址,所以隐去具体地址。

公网地址可参考社区版教程内容。

  wget http://xxx/repo/last/<配置文件>
  yum makecache

环境设置

内核参数修改

修改配置文件:vi /etc/sysctl.conf

添加以下内容

net.core.somaxconn = 2048
net.core.netdev_max_backlog = 10000
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

net.ipv4.ip_local_port_range = 3500 65535
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_slow_start_after_idle=0

vm.swappiness = 0
vm.min_free_kbytes = 2097152
vm.max_map_count=655360
fs.aio-max-nr=1048576

让配置生效

sysctl -p

会话变量修改

OceanBase 数据库的进程涉及的限制包括线程最大栈空间大小(Stack)、最大文件句柄数(Open Files)和 core 文件大小 (Core File Size)。

修改配置文件:vi /etc/security/limits.conf

  • 在全局级别修改。注意修改后,已经登录的会话需要退出重登录才生效。

将会话级别的最大栈空间大小设置为 unlimited,最大文件句柄数设置为 655350,Core 文件大小设置为 unlimited 。 修改 /etc/security/limits.conf 配置文件,如果已有设置值低于这个设置值

* soft nofile 655360
* hard nofile 655360
* soft nproc 655360
* hard nproc 655360
* soft core unlimited
* hard core unlimited
* soft stack unlimited
* hard stack unlimited

查看配置方法。退出当前会话,重新登录。执行以下命令,查看配置是否生效:

ulimit -a

关闭 SELinux

修改 SELinux 配置文件中的 SELINUX 选项。 注意:必须使用注释中的三个值之一。如果写错了,机器重启后操作系统会报错起不来,那时候就只能进入单用户模式修改了。

修改配置文件:vi /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled

配置文件修改后只会重启主机后生效,还需要使用下面命令立即生效。

setenforce 0

关闭防火墙

查看防火墙状态

systemctl status firewalld

如果是 inactive 那就不用管。如果是 active,那就永久关闭

systemctl disable firewalld 
systemctl stop firewalld
systemctl status firewalld

判断是否使用 ntpd同步时间。

systemctl status ntpd
Unit ntpd.service could not be found.

如果提示上面这个信息,表示没有使用 ntpd。 如果提示有 ntpd 服务,就卸载 ntpd 软件,检查chrony服务。

chrony服务检查

  • 查看时间同步活动

    chronyc activity
    
  • 查看时间服务器

    chronyc sources
    
  • 查看同步状态

    chronyc sources -v
    
  • 校准时间服务器:

    chronyc tracking
    

安装chrony服务

如果没有安装,可以采用 YUM 安装方法。您也可以下载相应的 RPM 包安装。

我的服务器已经安装,以下安装方法参考社区教程,仅供参考:

yum -y install chrony

chrony 配置说明

chrony 服务守护进程名是 chronydchronyc 是用来监控 chronyd 性能和配置参数的命令行工具。 chrony 的主配置文件: /etc/chrony.conf 。配置方法如下:

vi /etc/chrony.conf

# server 后面跟时间同步服务器
# 使用pool.ntp.org 项目中的公共服务器。按 server 配置,理论上您想添加多少时间服务器都可以。
# 或者使用 阿里云的 ntp 服务器
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server ntp.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst
server ntp.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp1.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp1.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst
server ntp10.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst

# 如果是测试环境,没有时间同步服务器,那就选取一台配置为时间同步服务器。
# 如果选中的是本机,则取消下面 server 注释
#server 127.127.1.0

# 根据实际时间计算出服务器增减时间的比率,然后记录到一个文件中,在系统重启后为系统做出最佳时间补偿调整。
driftfile /var/lib/chrony/drift

# chronyd 根据需求减慢或加速时间调整,
# 在某些情况下系统时钟可能漂移过快,导致时间调整用时过长。
# 该指令强制 chronyd 调整时期,大于某个阀值时步进调整系统时钟。
# 只有在因 chronyd 启动时间超过指定的限制时(可使用负值来禁用限制)没有更多时钟更新时才生效。
makestep 1.0 3

# 将启用一个内核模式,在该模式中,系统时间每11分钟会拷贝到实时时钟(RTC)。
rtcsync

# Enable hardware timestamping on all interfaces that support it.
# 通过使用hwtimestamp指令启用硬件时间戳
#hwtimestamp eth0
#hwtimestamp eth1
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

# 指定一台主机、子网,或者网络以允许或拒绝NTP连接到扮演时钟服务器的机器
#allow 192.168.0.0/16
#deny 192.168/16

# 即使没有同步到时间源,也要服务时间
local stratum 10

# 指定包含NTP验证密钥的文件。
#keyfile /etc/chrony.keys

# 指定日志文件的目录。
logdir /var/log/chrony



# Select which information is logged.
#log measurements statistics tracking

最简单的配置文件如下:

server 127.127.1.0
allow 172.20.0.0/16
local stratum 10

配置安装用户

  • 添加普通用户
groupadd -g 2000 admin 
useradd -u 2000 -g 2000 admin 
  • passwd admin

    输入密码即可

  • 添加admin的sudo权限

[root@localhost ~]# visudo

在以下内容后添加内容:

## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL  

添加以下内容:

admin ALL=(ALL) ALL

初始化目录

查看磁盘分区情况
[root@localhost ~]# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0   1.8T  0 disk 
sdb               8:16   0   1.8T  0 disk 
sdc               8:32   0 446.1G  0 disk 
├─sdc1            8:33   0     1G  0 part /boot/efi
├─sdc2            8:34   0     1G  0 part /boot
└─sdc3            8:35   0 444.1G  0 part 
  ├─vg00-lvroot 253:0    0   100G  0 lvm  /
  └─vg00-lvswap 253:1    0    16G  0 lvm  [SWAP]
loop0             7:0    0   387M  0 loop /usr/hxm/loadrunner
查看磁盘分区和挂载情况
[root@localhost ~]# lsblk -f
NAME            FSTYPE      LABEL       UUID                                   MOUNTPOINT
sda             LVM2_member             4bEcx7-m6bx-mL41-vKEO-DvOC-zEgE-H32Add 
sdb                                                                            
sdc                                                                            
├─sdc1          vfat                    0320-6ED4                              /boot/efi
├─sdc2          xfs                     93739afe-f19e-436e-aefb-be9522931fdf   /boot
└─sdc3          LVM2_member             22LfZ1-VwqX-MMKK-97L0-1l0T-iDlG-YPXHZ5 
  ├─vg00-lvroot xfs         lvroot      c2918725-45ec-4415-a7c1-d10598630178   /
  └─vg00-lvswap swap        lvswap      b9ce8c9c-03b2-4ceb-9044-945493566db5   [SWAP]
loop0           udf         T7330-15010 2010-08-25-10-57-00-0                  /usr/hxm/loadrunner
查看磁盘使用情况
[root@localhost ~]# df -hl
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 126G     0  126G   0% /dev
tmpfs                    126G  4.0K  126G   1% /dev/shm
tmpfs                    126G  116M  126G   1% /run
tmpfs                    126G     0  126G   0% /sys/fs/cgroup
/dev/mapper/vg00-lvroot  100G   43G   58G  43% /
/dev/sdc2               1014M  227M  788M  23% /boot
/dev/sdc1               1022M   12M 1011M   2% /boot/efi
tmpfs                     26G   12K   26G   1% /run/user/42
tmpfs                     26G     0   26G   0% /run/user/0
/dev/loop0               386M  386M     0 100% /usr/hxm/loadrunner
查看系统支持的分区类型

mkfs然后敲两下tab键查看我们的系统支持的分区类型

[root@localhost ~]# mkfs
mkfs         mkfs.btrfs   mkfs.cramfs  mkfs.ext2    mkfs.ext3    mkfs.ext4    mkfs.fat     mkfs.minix   mkfs.msdos   mkfs.vfat    mkfs.xfs
格式化分区
[root@localhost ~]# mkfs -t ext4 /dev/sda
mke2fs 1.42.9 (28-Dec-2013)
/dev/sda is entire device, not just one partition!
Proceed anyway? (y,n) y
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
117211136 inodes, 468843606 blocks
23442180 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2617245696
14308 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
        102400000, 214990848

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done     

[root@localhost ~]# mkfs -t ext4 /dev/sdb
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb is entire device, not just one partition!
Proceed anyway? (y,n) y
Discarding device blocks: done                            
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
117211136 inodes, 468843606 blocks
23442180 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2617245696
14308 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
        102400000, 214990848

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
查看磁盘分区和挂载情况
[root@localhost ~]# lsblk -f
NAME            FSTYPE      LABEL       UUID                                   MOUNTPOINT
sda             ext4                    90551db9-8abf-4b2e-80f7-27cb6d3139ce   
sdb             ext4                    31120181-6143-4385-bc62-8841e21941db   
sdc                                                                            
├─sdc1          vfat                    0320-6ED4                              /boot/efi
├─sdc2          xfs                     93739afe-f19e-436e-aefb-be9522931fdf   /boot
└─sdc3          LVM2_member             22LfZ1-VwqX-MMKK-97L0-1l0T-iDlG-YPXHZ5 
  ├─vg00-lvroot xfs         lvroot      c2918725-45ec-4415-a7c1-d10598630178   /
  └─vg00-lvswap swap        lvswap      b9ce8c9c-03b2-4ceb-9044-945493566db5   [SWAP]
loop0           udf         T7330-15010 2010-08-25-10-57-00-0                  /usr/hxm/loadrunner
目录创建
[root@localhost ~]# mkdir -p /data/1
[root@localhost ~]# mkdir -p /data/2
目录挂载

mount /dev/sda /data/1

这句话的意思就是把本设备的sda分区挂载到data目录下的1目录上

[root@localhost ~]# mount /dev/sda /data/1
[root@localhost ~]# mount /dev/sdb /data/2
查看磁盘分区和挂载情况

可以看到sda分区和sdb分区已经分别被成功的被挂载到/data/1he /data/2目录上了。

[root@localhost ~]# lsblk -f
NAME            FSTYPE      LABEL       UUID                                   MOUNTPOINT
sda             ext4                    90551db9-8abf-4b2e-80f7-27cb6d3139ce   /data/1
sdb             ext4                    31120181-6143-4385-bc62-8841e21941db   /data/2
sdc                                                                            
├─sdc1          vfat                    0320-6ED4                              /boot/efi
├─sdc2          xfs                     93739afe-f19e-436e-aefb-be9522931fdf   /boot
└─sdc3          LVM2_member             22LfZ1-VwqX-MMKK-97L0-1l0T-iDlG-YPXHZ5 
  ├─vg00-lvroot xfs         lvroot      c2918725-45ec-4415-a7c1-d10598630178   /
  └─vg00-lvswap swap        lvswap      b9ce8c9c-03b2-4ceb-9044-945493566db5   [SWAP]
loop0           udf         T7330-15010 2010-08-25-10-57-00-0                  /usr/hxm/loadrunner
检查挂载是否成功
[root@localhost /]# ls /data/1
lost+found
[root@localhost /]# ls /data/2
lost+found

出现lost+found 就说明我们挂载成功了。

设置可以自动挂载(永久挂载)

检查当前挂载情况:

[root@localhost /]# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Sat Dec 18 00:52:03 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg00-lvroot /                       xfs     defaults        0 0
UUID=93739afe-f19e-436e-aefb-be9522931fdf /boot                   xfs     defaults        0 0
UUID=0320-6ED4          /boot/efi               vfat    defaults,uid=0,gid=0,umask=0077,shortname=winnt 0 0
/dev/mapper/vg00-lvswap swap                    swap    defaults        0 0

后我们不用UUID,直接写清楚把这台设备的哪个分区挂载到哪个目录下:

输入 vim /etc/fstab,进入修改界面:

将/data/1和data/2进行填写

#
# /etc/fstab
# Created by anaconda on Sat Dec 18 00:52:03 2021
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg00-lvroot /                       xfs     defaults        0 0
UUID=93739afe-f19e-436e-aefb-be9522931fdf /boot                   xfs     defaults        0 0
UUID=0320-6ED4          /boot/efi               vfat    defaults,uid=0,gid=0,umask=0077,shortname=winnt 0 0
/dev/mapper/vg00-lvswap swap                    swap    defaults        0 0
/dev/sda /data/1                                ext4    defaults        0 0
/dev/sdb /data/2                                ext4    defaults        0 0

这个意思就是把这台机器的sda分区挂载到/data/1,sdb分区挂载到/data/2,默认我们改成0 0就行。

保存退出,然后执行mount -a指令(a指auto,意思是让它自动挂载)

[root@localhost /]# mount -a
[root@localhost /]# 

然后就完成自动挂载了。

更改目录权限
[root@localhost /]# chown -R admin.admin /data/1 && chown -R admin.admin /data/2
[root@localhost /]# 

OceanBase下载和安装

软件下载

在官网下载社区版最新应用包

obclient-2.0.0-2.el7.x86_64.rpm
libobclient-2.0.0-2.el7.x86_64.rpm
obproxy-3.2.0-1.el7.x86_64.rpm
oceanbase-ce-3.1.2-10000392021123010.el7.x86_64.rpm
oceanbase-ce-libs-3.1.2-10000392021123010.el7.x86_64.rpm

安装

目录规划
目录规划:

/home/admin/oceanbase   --observer部署启动目录   RPM包自动创建
/home/admin/oceanbase/ob  observer数据总目录  手动创建
/data/1/{observer01,observer02,observer03,obproxy}  --obs数据文件实际目录   手动创建连接至数据总目录
/data/2/observer{01,02,03}/store/{sort_dir,sstable,clog,ilog,slog}  --事务日志实际目录   手动创建连接至数据总目录
/home/admin/oceanbase/etc  --配置文件 启动时在启动目录自动创建
/home/admin/oceanbase/log   --运行日志目录 启动时在启动目录自动创建

按照规划目录创建对应目录

安装

使用root用户执行安装:

安装oceanbase依赖
[root@localhost 3.1.2_ce]# rpm -ivh oceanbase-ce-libs-3.1.2-10000392021123010.el7.x86_64.rpm 
warning: oceanbase-ce-libs-3.1.2-10000392021123010.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e9b4a7aa: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:oceanbase-ce-libs-3.1.2-100003920################################# [100%]
安装oceanbase
[root@localhost 3.1.2_ce]# rpm -ivh oceanbase-ce-3.1.2-10000392021123010.el7.x86_64.rpm 
warning: oceanbase-ce-3.1.2-10000392021123010.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e9b4a7aa: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:oceanbase-ce-3.1.2-10000392021123################################# [100%]
安装obproxy
[root@localhost 3.1.2_ce]# rpm -ivh obproxy-3.2.0-1.el7.x86_64.rpm 
warning: obproxy-3.2.0-1.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e9b4a7aa: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:obproxy-3.2.0-1.el7              ################################# [100%]
安装libobclient
[root@localhost 3.1.2_ce]# rpm -ivh libobclient-2.0.0-2.el7.x86_64.rpm 
warning: libobclient-2.0.0-2.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e9b4a7aa: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:libobclient-2.0.0-2.el7          ################################# [100%]
[root@localhost 3.1.2_ce]# rpm -ivh obclient-2.0.0-2.el7.x86_64.rpm 
warning: obclient-2.0.0-2.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e9b4a7aa: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:obclient-2.0.0-2.el7             ################################# [100%]
查看安装后信息
[root@localhost 3.1.2_ce]# rpm -ql oceanbase-ce
/home/admin/oceanbase/bin
/home/admin/oceanbase/bin/import_time_zone_info.py
/home/admin/oceanbase/bin/observer
/home/admin/oceanbase/etc
/home/admin/oceanbase/etc/oceanbase_upgrade_dep.yml
/home/admin/oceanbase/etc/priv_checker.py
/home/admin/oceanbase/etc/timezone_V1.log
/home/admin/oceanbase/etc/upgrade_checker.py
/home/admin/oceanbase/etc/upgrade_cluster_health_checker.py
/home/admin/oceanbase/etc/upgrade_post.py
/home/admin/oceanbase/etc/upgrade_post_checker.py
/home/admin/oceanbase/etc/upgrade_pre.py
/home/admin/oceanbase/etc/upgrade_rolling_post.py
/home/admin/oceanbase/etc/upgrade_rolling_pre.py
[root@localhost 3.1.2_ce]# rpm -ql oceanbase-ce-libs
/home/admin/oceanbase/lib
/home/admin/oceanbase/lib/libaio.so
/home/admin/oceanbase/lib/libaio.so.1
/home/admin/oceanbase/lib/libaio.so.1.0.1
/home/admin/oceanbase/lib/libmariadb.so
/home/admin/oceanbase/lib/libmariadb.so.3
[root@localhost 3.1.2_ce]# rpm -ql obproxy
/home/admin/obproxy-3.2.0/bin
/home/admin/obproxy-3.2.0/bin/obproxy
/home/admin/obproxy-3.2.0/bin/obproxyd.sh
root@localhost 3.1.2_ce]# rpm -ql obclient
/u01/obclient
/u01/obclient/bin
/u01/obclient/bin/aria_chk
/u01/obclient/bin/aria_dump_log
/u01/obclient/bin/aria_ftdump
/u01/obclient/bin/aria_pack
/u01/obclient/bin/aria_read_log
/u01/obclient/bin/innochecksum
/u01/obclient/bin/mariabackup
/u01/obclient/bin/mbstream
/u01/obclient/bin/msql2mysql
/u01/obclient/bin/my_print_defaults
/u01/obclient/bin/myisam_ftdump
/u01/obclient/bin/myisamchk
/u01/obclient/bin/myisamlog
/u01/obclient/bin/myisampack
/u01/obclient/bin/mysql_config
/u01/obclient/bin/mysql_convert_table_format
/u01/obclient/bin/mysql_find_rows
/u01/obclient/bin/mysql_fix_extensions
/u01/obclient/bin/mysql_plugin
/u01/obclient/bin/mysql_secure_installation
/u01/obclient/bin/mysql_setpermission
/u01/obclient/bin/mysql_tzinfo_to_sql
/u01/obclient/bin/mysql_upgrade
/u01/obclient/bin/mysql_waitpid
/u01/obclient/bin/mysqlaccess
/u01/obclient/bin/mysqladmin
/u01/obclient/bin/mysqlbinlog
/u01/obclient/bin/mysqlcheck
/u01/obclient/bin/mysqld
/u01/obclient/bin/mysqld_multi
/u01/obclient/bin/mysqld_safe
/u01/obclient/bin/mysqld_safe_helper
/u01/obclient/bin/mysqldump
/u01/obclient/bin/mysqldumpslow
/u01/obclient/bin/mysqlhotcopy
/u01/obclient/bin/mysqlimport
/u01/obclient/bin/mysqlshow
/u01/obclient/bin/mysqlslap
/u01/obclient/bin/mysqltest
/u01/obclient/bin/mytop
/u01/obclient/bin/obclient
/u01/obclient/bin/perror
/u01/obclient/bin/replace
/u01/obclient/bin/resolve_stack_dump
/u01/obclient/bin/resolveip
/u01/obclient/bin/wsrep_sst_common
/u01/obclient/bin/wsrep_sst_mariabackup
/u01/obclient/bin/wsrep_sst_mysqldump
/u01/obclient/bin/wsrep_sst_rsync
/u01/obclient/bin/wsrep_sst_rsync_wan
启动OB进程
配置lib信息
[admin@localhost soft]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/admin/oceanbase/lib/
[admin@localhost soft]$ 

也可以将以上内容写入到admin用户的.bash_profile中,然后执行source ~/.bash_profile生效。

查询网卡名称

确认网卡名称:bond0

[admin@localhost observer03]$ ifconfig
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST>  mtu 1500
        inet 10.144.2.112  netmask 255.255.255.0  broadcast 10.144.2.255
        inet6 fe80::e42:a1ff:fe25:edc2  prefixlen 64  scopeid 0x20<link>
        ether 0c:42:a1:25:ed:c2  txqueuelen 1000  (Ethernet)
        RX packets 5097515761  bytes 3604099003530 (3.2 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5077664702  bytes 817770325512 (761.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::42:29ff:feb6:f680  prefixlen 64  scopeid 0x20<link>
        ether 02:42:29:b6:f6:80  txqueuelen 0  (Ethernet)
        RX packets 8  bytes 544 (544.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16  bytes 1978 (1.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 38:68:dd:3b:da:48  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 38:68:dd:3b:da:49  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s20f0u1u6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 3a:68:dd:3b:da:4f  txqueuelen 1000  (Ethernet)
        RX packets 2763517  bytes 194176501 (185.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens2f0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 0c:42:a1:25:ed:c2  txqueuelen 1000  (Ethernet)
        RX packets 5097515894  bytes 3604099116132 (3.2 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5077597865  bytes 817762040728 (761.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens2f1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST>  mtu 1500
        ether 0c:42:a1:25:ed:c2  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 66988  bytes 8306512 (7.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 385056279  bytes 38294702923 (35.6 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 385056279  bytes 38294702923 (35.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:b4:e2:95  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
启动OB进程
cd /data/1/observer01/ && /home/admin/oceanbase/bin/observer -r "10.144.2.112:2882:2881;10.144.2.112:3882:3881;10.144.2.112:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/2/observer01/store -i bond0 -l INFO
cd /data/1/observer02/ && /home/admin/oceanbase/bin/observer -r "10.144.2.112:2882:2881;10.144.2.112:3882:3881;10.144.2.112:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/2/observer02/store -i bond0 -l INFO
cd /data/1/observer03/ && /home/admin/oceanbase/bin/observer -r "10.144.2.112:2882:2881;10.144.2.112:3882:3881;10.144.2.112:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/2/observer03/store -i bond0 -l INFO

以上命令中指定的参数作用可参考obd自动部署过程yaml文件的说明,请移步 https://gitee.com/oceanbase/obdeploy/tree/master/example

查看进程信息
[admin@localhost observer03]$ ps -ef|grep observer
admin    230427      1 72 01:48 ?        00:00:26 /home/admin/oceanbase/bin/observer -r 10.144.2.112:2882:2881;10.144.2.112:3882:3881;10.144.2.112:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/2/observer01/store -i bond0 -l INFO
admin    231490      1 86 01:48 ?        00:00:12 /home/admin/oceanbase/bin/observer -r 10.144.2.112:2882:2881;10.144.2.112:3882:3881;10.144.2.112:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/2/observer02/store -i bond0 -l INFO
admin    232229      1 97 01:48 ?        00:00:06 /home/admin/oceanbase/bin/observer -r 10.144.2.112:2882:2881;10.144.2.112:3882:3881;10.144.2.112:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/2/observer03/store -i bond0 -l INFO
admin    232926  84741  0 01:48 pts/1    00:00:00 grep --color=auto observer
集群初始化
连接OceanBase

默认密码为空,直接回车即可。

mysql -h10.144.2.112 -uroot -P2881 -p -c -A
设置超时时间
MySQL [(none)]> set session ob_query_timeout=1000000000;
Query OK, 0 rows affected (0.00 sec)
将3个zone初始化为一个集群
MySQL [(none)]> alter system bootstrap ZONE 'zone1' SERVER '10.144.2.112:2882' , ZONE 'zone2' SERVER '10.144.2.112:3882' , ZONE 'zone3' SERVER '10.144.2.112:4882';
Query OK, 0 rows affected (22.33 sec)
修改root密码
MySQL [(none)]> alter user root identified by 'root';
Query OK, 0 rows affected (0.03 sec)
初始化登录
mysql -h127.1 -uroot@sys -P2881 -proot -c -A oceanbase
创建OBProxy用户
  • OBPROXY 用户(proxyro)密码默认 OBPROXY 连接 OceanBase 集群使用用户 proxyro 。这个用户不存在,需要创建。
MySQL [oceanbase]> create user proxyro identified by 'proxyro';
Query OK, 0 rows affected (0.04 sec)

MySQL [oceanbase]> grant select on oceanbase.* to proxyro;
Query OK, 0 rows affected (0.02 sec)
启动OBProxy
启动OBProxy

进入obproxy安装目录

[admin@localhost observer03]$ cd /data/1/obproxy/

执行启动命令

[admin@localhost obproxy]$ /home/admin/obproxy-3.2.0/bin/obproxy -p2883 -c obcluster -r "10.144.2.112:2881;10.144.2.112:3881;10.144.2.112:4881" -o "enable_cluster_checkout=false,enable_strict_kernel_release=false,enable_metadb_used=false"
/home/admin/obproxy-3.2.0/bin/obproxy -p2883 -c obcluster -r 10.144.2.112:2881;10.144.2.112:3881;10.144.2.112:4881 -o enable_cluster_checkout=false,enable_strict_kernel_release=false,enable_metadb_used=false
listen port: 2883
cluster_name: obcluster
rs list: 10.144.2.112:2881;10.144.2.112:3881;10.144.2.112:4881
optstr: enable_cluster_checkout=false,enable_strict_kernel_release=false,enable_metadb_used=false
检查 OBPROXY 监听正常

进程 obproxy 默认会监听2个端口:2883 和 2884 。

[admin@localhost log]$ netstat -ntlp |grep obproxy
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:2883            0.0.0.0:*               LISTEN      261142/obproxy      
tcp        0      0 0.0.0.0:2884            0.0.0.0:*               LISTEN      261142/obproxy
登录 OBPROXY 修改密码

登录用户名:root@proxysys, 端口:2883 ,初始密码:空。

[admin@localhost log]$ mysql -h 10.144.2.112 -u root@proxysys -P 2883 -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.6.25

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>show proxyconfig like '%sys_password%';
+------------------------+-------+--------------------------------+-------------+---------------+
| name                   | value | info                           | need_reboot | visible_level |
+------------------------+-------+--------------------------------+-------------+---------------+
| observer_sys_password1 |       | password for observer sys user | false       | SYS           |
| observer_sys_password  |       | password for observer sys user | false       | SYS           |
| obproxy_sys_password   |       | password for obproxy sys user  | false       | SYS           |
+------------------------+-------+--------------------------------+-------------+---------------+
3 rows in set (0.00 sec)

修改 OBPROXY 用户密码是通过修改参数的方式,命令是:alter proxyconfig set

MySQL [(none)]> alter proxyconfig set obproxy_sys_password = 'proxyconfig' ;
Query OK, 0 rows affected (0.00 sec)

同时还需要修改 OBPROXY 连接 OceanBase 集群用户 proxyro 的密码。这样 OBPROXY 才能跟 OceanBase 集群正常连接。这个密码就是前面 OceanBase 集群初始化后创建的用户 proxyro 的密码。

MySQL [(none)]> alter proxyconfig set observer_sys_password = 'proxyro' ;
Query OK, 0 rows affected (0.00 sec)

退出,通过 OBPROXY 连接 OceanBase 集群看看, 如果能查看所有会话,则说明 OBPROXY 部署成功。

[admin@localhost ~]$ mysql -h127.1 -uroot@sys#obcluster -P2883 -pproxyro -c -A oceanbase
ERROR 1045 (42000): Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: YES)
[admin@localhost ~]$ mysql -h127.1 -uroot@sys#obcluster -P2883 -proot -c -A oceanbase
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 41
Server version: 5.6.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [oceanbase]> show processlist;
+------+--------+------+-----------------+-----------+-------------+-------------------+-------------------+--------+--------+
| Id   | Tenant | User | Host            | db        | trans_count | svr_session_count | state             | tid    | pid    |
+------+--------+------+-----------------+-----------+-------------+-------------------+-------------------+--------+--------+
|   41 | sys    | root | 127.0.0.1:22552 | oceanbase |           0 |                 1 | MCS_ACTIVE_READER | 261142 | 261142 |
+------+--------+------+-----------------+-----------+-------------+-------------------+-------------------+--------+--------+
1 row in set (0.00 sec)

MySQL [oceanbase]> 

创建业务租户、业务数据库、测试表

mysql -h127.1 -uroot@sys -P2881 -proot -c -A oceanbase

查询系统资源总计资源

SELECT svr_ip,svr_port, cpu_total, mem_total/1024/1024/1024, disk_total/1024/1024/1024, zone FROM __all_virtual_server_stat;

查询租户已分配资源:

SELECT sum(c.max_cpu), sum(c.max_memory)/1024/1024/1024 FROM __all_resource_pool as a, __all_unit_config AS c WHERE a.unit_config_id=c.unit_config_id;

创建租户

创建资源单元
MySQL [oceanbase]> CREATE resource unit u1 max_cpu=4, min_cpu=4, max_memory='1G', min_memory='1G', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='100G';
Query OK, 0 rows affected (0.01 sec)
查看新创建的资源单元:
MySQL [oceanbase]> SELECT unit_config_id,name,max_cpu,min_cpu,max_memory,min_memory,max_disk_size FROM __all_unit_config;
+----------------+-----------------+---------+---------+------------+------------+---------------+
| unit_config_id | name            | max_cpu | min_cpu | max_memory | min_memory | max_disk_size |
+----------------+-----------------+---------+---------+------------+------------+---------------+
|              1 | sys_unit_config |       5 |     2.5 | 1288490188 | 1073741824 |   53687091200 |
|           1001 | u1              |       4 |       4 | 1073741824 | 1073741824 |  107374182400 |
+----------------+-----------------+---------+---------+------------+------------+---------------+
2 rows in set (0.00 sec)
创建资源池
MySQL [oceanbase]> CREATE resource pool my_pool unit = 'u1', unit_num = 1;
Query OK, 0 rows affected (0.01 sec)
创建租户
MySQL [oceanbase]> create tenant obmysql3 resource_pool_list=('my_pool'), primary_zone='zone1,zone2,zone3',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%', ob_compatibility_mode='mysql';
Query OK, 0 rows affected (1.20 sec)
连接白名单设置

需要设置白名单否则只能127.0.0.1登录,需要在sys租户下设置

MySQL [oceanbase]> ALTER TENANT obmysql3 SET VARIABLES ob_tcp_invited_nodes='%';
Query OK, 0 rows affected (0.003 sec)
查看创建成功的租户:
MySQL [oceanbase]> select tenant_id,tenant_name,primary_zone from __all_tenant;
+-----------+-------------+-------------------+
| tenant_id | tenant_name | primary_zone      |
+-----------+-------------+-------------------+
|         1 | sys         | zone1;zone2,zone3 |
|      1001 | obmysql3    | zone1,zone2,zone3 |
+-----------+-------------+-------------------+
2 rows in set (0.01 sec)

创建业务用户

创建用户
MySQL [oceanbase]> create user testuser2@'%' identified by 'testuser2' ;
Query OK, 0 rows affected (0.029 sec)
查看创建成功的用户
MySQL [oceanbase]> SELECT user FROM mysql.user;
+------------+
| user       |
+------------+
| root       |
| ORAAUDITOR |
| testuser2  |
+------------+
3 rows in set (0.043 sec)
授权
MySQL [oceanbase]> grant all privileges on *.* to testuser2@'%';
Query OK, 0 rows affected (0.025 sec)
查看授权
MySQL [oceanbase]> show grants for testuser2;
+--------------------------------------------+
| Grants for testuser2@%                     |
+--------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'testuser2' |
+--------------------------------------------+
1 row in set (0.019 sec)

创建业务数据库

新建用户进行登录

退出sys租户登录,使用新建业务用户和租户登录。

[admin@localhost ~]$ obclient -h127.1 -utestuser2@obmysql3#obcluster  -P2883 -ptestuser2 -c -A oceanbase
Welcome to the OceanBase.  Commands end with ; or \g.
Your MySQL connection id is 49
Server version: 5.6.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [oceanbase]> 
创建业务库
MySQL [oceanbase]> create database testdb2 ;
Query OK, 1 row affected (0.021 sec)

创建业务表

新建业务租户登录

退出sys租户登录,使用新建业务租户登录。

[admin@localhost ~]$ obclient -h127.1 -utestuser2@obmysql3#obcluster  -P2883 -ptestuser2 -c -A oceanbase
Welcome to the OceanBase.  Commands end with ; or \g.
Your MySQL connection id is 49
Server version: 5.6.25 OceanBase 3.1.2 (r10000392021123010-d4ace121deae5b81d8f0b40afbc4c02705b7fc1d) (Built Dec 30 2021 02:47:29)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [oceanbase]> 
创建表
MySQL [oceanbase]> use testdb2;
Database changed
MySQL [testdb2]> DROP TABLE IF EXISTS B;
Query OK, 0 rows affected, 1 warning (0.004 sec)

MySQL [testdb2]> CREATE TABLE B (
    ->   INTERNAL_KEY BIGINT NOT NULL ,
    ->   BASE_ACCT_NO VARCHAR(50)  NOT NULL ,
    ->   PROD_TYPE VARCHAR(50)  DEFAULT NULL ,
    ->   CCY VARCHAR(3)  DEFAULT NULL ,
    ->   ACCT_SEQ_NO VARCHAR(8)  DEFAULT NULL ,
    ->   BRANCH VARCHAR(20)  DEFAULT NULL ,
    ->   CLIENT_NO VARCHAR(20)  DEFAULT NULL ,
    ->   ACCT_OPEN_DATE VARCHAR(8)  DEFAULT NULL ,
    ->   BAL_TYPE VARCHAR(2)  DEFAULT NULL ,
    ->   TERM VARCHAR(5)  DEFAULT NULL ,
    ->   MATURITY_DATE VARCHAR(8)  DEFAULT NULL ,
    ->   AUTO_RENEW_ROLLOVER VARCHAR(1)  DEFAULT NULL ,
    ->   USER_ID VARCHAR(30)  DEFAULT NULL ,
    ->   LAST_CHANGE_DATE VARCHAR(8)  DEFAULT NULL ,
    -> CMISLOAN_NO VARCHAR(50)  DEFAULT NULL,
    -> PRIMARY KEY (BASE_ACCT_NO,INTERNAL_KEY)
    -> );
Query OK, 0 rows affected (0.087 sec)

操作业务表

MySQL [testdb2]> insert into b  values (2,'2','110','CNY','1','999999','2','20220113','T','Q','20220113','2','2','20220113','2');
Query OK, 1 row affected (0.002 sec)

卸载

[admin ~]$ kill -9 `pidof observer` `pidof obproxy`
[admin ~]$ bin/rm -rf /data/{1,2}/observer{01,02,03} /data/1/obproxy

部署过程问题列表

  1. 部署OBProxy后,使用系统租户连接报错

    [admin@localhost obproxy]$ mysql -h127.1 -uroot@sys -P2883 -p -c -A oceanbase
    Enter password: 
    ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0
    

    问题原因:

    未创建proxyro用户,并且还可能未对proxyconfig的密码进行修改。

    需要提前创建proxyro用户。

  2. 部署OceanBase后,启动命令执行成功,但查询不到进程。并且observer.log中有error日志错误

    init config fail(ret=4002)

    问题原因:启动命令中有不符合规范的符号,必须分号或者冒号,导致配置初始化异常。启动失败。

    当前启动OceanBase并不会在启动命令执行后就报错,需要自己在observer.log中找对应的日志,可以选择第一个出行的error日志,基本就能找到具体原因。

  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值