Apache Kudu 1.15.0的分布式集群部署

1. 部署规划

服务	服务器	备注
master	kudu1、kudu2、kudu3
tablet server	kudu1、kudu2、kudu3

2. 部署要求

2.1 硬件要求

1. 必须是Centos6.4之后的版本才支持Hole Punching，可以通过如下方式进行查看

[root@kudu1 ~]# uname -r
3.10.0-1160.el7.x86_64
[root@kudu1 ~]#

2. 必须是ext4或xfs类型的文件系统才支持Hole Punching，可以通过以下方式进行查看

[root@kudu1 ~]# df -hT
文件系统                       类型      容量  已用  可用 已用% 挂载点
devtmpfs                       devtmpfs  475M     0  475M    0% /dev
tmpfs                          tmpfs     487M     0  487M    0% /dev/shm
tmpfs                          tmpfs     487M  7.7M  479M    2% /run
tmpfs                          tmpfs     487M     0  487M    0% /sys/fs/cgroup
/dev/mapper/centos_centos-root xfs        50G  1.4G   49G    3% /
/dev/mapper/centos_centos-home xfs        47G   33M   47G    1% /home
/dev/sda1                      xfs      1014M  138M  877M   14% /boot
tmpfs                          tmpfs      98M     0   98M    0% /run/user/0
[root@kudu1 ~]# 

有时查看的文件系统是ext4格式，但其实是ext2、ext3的设备格式化的，也是不支持Hole Punching的。下面是一个查看磁盘分区属于哪个ext版本的方法，仅供参考

[root@kudu1 ~]# file -sL /dev/sd*
/dev/sda1: Linux rev 1.0 ext4 filesystem data, UUID=63fa0104-4aab-4dc8-a50d-e2c1bf0fb188 (extents) (large files) (huge files)
/dev/sdb1: Linux rev 1.0 ext2 filesystem data, UUID=b3c82023-78e1-4ad4-b6e0-62355b272166
/dev/sdb2: Linux/i386 swap file (new style), version 1 (4K pages), size 4194303 pages, no label, UUID=3f64308c-19db-4da5-a9a0-db4d7defb80f
[root@kudu1 ~]# 

3. Kudu集群的各个服务器之间的时间必须保持同步。参考Centos7服务器通过Chrony设置时间同步

2.2 软件要求

每台服务器都安装Java8，参考centos7同时安装java8和openJdk11、windows同时安装java8和openJdk11

2.3 安装依赖

[root@kudu1 ~]# yum install gcc cyrus-sasl-plain

3. 从源码编译(kudu1上操作)

3.1 安装Red Hat Developer Toolset

为了能够使用C++ 17编译器，需要安装Red Hat Developer Toolset

[root@kudu1 ~]#
[root@kudu1 ~]# yum install centos-release-scl-rh
[root@kudu1 ~]#
[root@kudu1 ~]# yum install devtoolset-8
[root@kudu1 ~]#

在/root/.bashrc添加如下内容，并执行source /root/.bashrc

source /opt/rh/devtoolset-8/enable

从下列命令结果可以看到，gcc已经是版本8了

[root@kudu1 ~]# 
[root@kudu1 ~]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-8/root/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-8/root/usr --mandir=/opt/rh/devtoolset-8/root/usr/share/man --infodir=/opt/rh/devtoolset-8/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --with-default-libstdcxx-abi=gcc4-compatible --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-8.3.1-20190311/obj-x86_64-redhat-linux/isl-install --disable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC) 
[root@kudu1 ~]#

3.2 安装依赖包

[root@kudu1 ~]# 
[root@kudu1 ~]# yum install autoconf automake cyrus-sasl-devel cyrus-sasl-gssapi \
> cyrus-sasl-plain flex gcc gcc-c++ gdb git java-1.8.0-openjdk-devel \
> krb5-server krb5-workstation libtool make openssl-devel patch \
> pkgconfig redhat-lsb-core rsync unzip vim-common which
[root@kudu1 ~]# 

3.3 安装memkind(可选)

如果需要Kudu的NVM(non-volatile memory)block cache支持，需要安装1.8.0以上的memkind

[root@kudu1 ~]# yum list memkind
已加载插件：fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * centos-sclo-rh: mirrors.cn99.com
 * epel: ftp.iij.ad.jp
 * extras: mirrors.163.com
 * updates: mirrors.163.com
可安装的软件包
memkind.x86_64                                                                       1.7.0-1.el7                                                                       base
[root@kudu1 ~]#

linux提供的版本是1.7.0，需要手动编译更高版本的

[root@kudu1 ~]# yum install numactl-libs numactl-devel
[root@kudu1 ~]#
[root@kudu1 ~]# wget -O memkind-1.13.0.tar.gz https://github.com/memkind/memkind/archive/refs/tags/v1.13.0.tar.gz
[root@kudu1 ~]# 
[root@kudu1 ~]# tar -zxvf memkind-1.13.0.tar.gz
[root@kudu1 ~]# 
[root@kudu1 ~]# cd memkind-1.13.0
[root@kudu1 memkind-1.13.0]# 
[root@kudu1 memkind-1.13.0]# ./autogen.sh
[root@kudu1 memkind-1.13.0]# 
[root@kudu1 memkind-1.13.0]# ./configure --prefix=/usr
[root@kudu1 memkind-1.13.0]#
[root@kudu1 memkind-1.13.0]# make
[root@kudu1 memkind-1.13.0]#
[root@kudu1 memkind-1.13.0]# make install
[root@kudu1 memkind-1.13.0]#
[root@kudu1 memkind-1.13.0]# man memkind
[root@kudu1 memkind-1.13.0]#

3.4 编译documentation的依赖包(可选)

暂不安装

3.5 编译Kudu

[root@kudu1 ~]#
[root@kudu1 ~]# wget -O kudu-1.15.0.tar.gz https://github.com/apache/kudu/archive/refs/tags/1.15.0.tar.gz
[root@kudu1 ~]#
[root@kudu1 ~]# tar -zxvf kudu-1.15.0.tar.gz
[root@kudu1 ~]#
[root@kudu1 ~]# cd kudu-1.15.0
[root@kudu1 kudu-1.15.0]#

编译第三方依赖包

[root@kudu1 kudu-1.15.0]# 
[root@kudu1 kudu-1.15.0]# build-support/enable_devtoolset.sh thirdparty/build-if-necessary.sh
......省略部分......
Running post-flight checks
-------------------------
Checking that no TSAN dependency depends on libstdc++ ... PASSED
-------------------------
Post-flight checks succeeded.
+ echo ---------------------
---------------------
+ echo 'Thirdparty dependencies '\''common uninstrumented'\'' built and installed successfully'
Thirdparty dependencies 'common uninstrumented' built and installed successfully
+ exit 0
[root@kudu1 kudu-1.15.0]#

编译kudu：创建一个编译目录作为中间输出目录(不能是kudu的根目录)

[root@kudu1 kudu-1.15.0]# 
[root@kudu1 kudu-1.15.0]# mkdir -p build/release
[root@kudu1 kudu-1.15.0]# 
[root@kudu1 kudu-1.15.0]# cd build/release
[root@kudu1 release]# 
[root@kudu1 release]# ../../build-support/enable_devtoolset.sh \
> ../../thirdparty/installed/common/bin/cmake \
> -DCMAKE_BUILD_TYPE=release ../..
......省略部分......
-- Configuring done
-- Generating done
-- Build files have been written to: /root/kudu-1.15.0/build/release
[root@kudu1 release]#
[root@kudu1 release]# make -j4
[root@kudu1 release]# 

这里如果编译报错误：已杀死 signal terminated program cc1plus，则需要添加服务器的内存资源。且安装目录下面需要150G的空余磁盘空间

安装executables可执行文件、libraries和headers(可选)

[root@kudu1 ~]# 
[root@kudu1 ~]# mkdir kudu-1.15.0-install
[root@kudu1 ~]#
[root@kudu1 ~]# cd kudu-1.15.0/build/release/
[root@kudu1 release]#
[root@kudu1 release]# make DESTDIR=/root/kudu-1.15.0-install install
[root@kudu1 release]#

如果不指定DESTDIR，则默认目录为/usr/local，会在DESTDIR目录下生成以下

• usr/local/sbin目录：kudu-tserver和kudu-master可执行文件
• usr/local/bin目录：kudu命令行工具
• usr/local/lib64目录：kudu的client library
• usr/local/include/kudu目录：kudu的client headers

编译documentation(可选)

暂不安装

4. 配置Kudu(kudu1上操作)

4.1 配置基础

kudu-master和kudu-tserver有2种方式传递配置参数

1. 直接传递配置参数，比如kudu-master --master_addresses=localhost --log_dir=/tmp
2. 传递一个或多个配置文件，比如kudu-master --flagfile=/tmp/master.gflagfile。配置文件中可以包含配置参数和配置文件

--master_addresses=localhost
--flagfile=/tmp/tmp.gflagfile

查看帮助信息，通过–help获取，比如kudu-master --help

4.2 拷贝kudu的webserver的CSS文件

[root@kudu1 local]# 
[root@kudu1 local]# pwd
/root/kudu-1.15.0-install/usr/local
[root@kudu1 local]# 
[root@kudu1 local]# scp -r /root/kudu-1.15.0/www .
[root@kudu1 local]# 

4.3 创建数据目录

[root@kudu1 local]#
[root@kudu1 local]# pwd
/root/kudu-1.15.0-install/usr/local
[root@kudu1 local]# 
[root@kudu1 local]# mkdir conf
[root@kudu1 local]# 
[root@kudu1 local]# mkdir -p data/master/data
[root@kudu1 local]# mkdir -p data/master/metadata
[root@kudu1 local]# mkdir -p data/master/wal
[root@kudu1 local]# mkdir -p data/master/log
[root@kudu1 local]# 
[root@kudu1 local]# mkdir -p data/tserver/data
[root@kudu1 local]# mkdir -p data/tserver/metadata
[root@kudu1 local]# mkdir -p data/tserver/wal
[root@kudu1 local]# mkdir -p data/tserver/log
[root@kudu1 local]# 

4.4 conf/master.gflagfile

新增master.gflagfile，文件内容如下：

# 所有的Master RPC地址，默认localhost
--master_addresses=kudu1:7051,kudu2:7051,kudu3:7051

# master储存data blocks的目录，多个逗号分隔
--fs_data_dirs=/root/kudu-1.15.0-install/usr/local/data/master/data

# master储存tablet metadata的目录
--fs_metadata_dir=/root/kudu-1.15.0-install/usr/local/data/master/metadata

# master储存write-ahead logs的目录
--fs_wal_dir=/root/kudu-1.15.0-install/usr/local/data/master/wal

# master储存log files的目录，默认/tmp
--log_dir=/root/kudu-1.15.0-install/usr/local/data/master/log

# 指定master的webserver的CSS文件
--webserver_doc_root=/root/kudu-1.15.0-install/usr/local/www

# 指定master的block cache大小，默认512
--block_cache_capacity_mb=128

# 允许任何服务器的IP地址进行访问master，默认为127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
--trusted_subnets=0.0.0.0/0

4.5 conf/tserver.gflagfile

新增tserver.gflagfile，文件内容如下：

# 连接的Master RPC地址，默认127.0.0.1:7051
--tserver_master_addrs=kudu1:7051,kudu2:7051,kudu3:7051

# tserver储存data blocks的目录，多个逗号分隔
--fs_data_dirs=/root/kudu-1.15.0-install/usr/local/data/tserver/data

# tserver储存tablet metadata的目录
--fs_metadata_dir=/root/kudu-1.15.0-install/usr/local/data/tserver/metadata

# tserver储存write-ahead logs的目录
--fs_wal_dir=/root/kudu-1.15.0-install/usr/local/data/tserver/wal

# tserver储存log files的目录，默认/tmp
--log_dir=/root/kudu-1.15.0-install/usr/local/data/tserver/log

# 分配给tablet server的block cache最大内存，默认512
--block_cache_capacity_mb=256

# tablet server能使用的最大内存，超过该值就拒绝write请求，默认4294967296
--memory_limit_hard_bytes=1073741824

# 指定master的webserver的CSS文件
--webserver_doc_root=/root/kudu-1.15.0-install/usr/local/www

# 允许任何服务器的IP地址进行访问tserver，默认为127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,169.254.0.0/16
--trusted_subnets=0.0.0.0/0

4.6 Tables配置

可以按表粒度设置，设置方式有2种

1. 创建表的时候设置
2. 通过API或命令行进行修改

Configuration	类型	默认值	描述
kudu.table.history_max_age_sec	integer		Number of seconds to retain history for tablets in this table
kudu.table.maintenance_priority	integer	0	Priority level of a table for maintenance

5. 分发kudu安装文件(kudu1上操作)

[root@kudu1 ~]# scp -r kudu-1.15.0-install root@kudu2:/root
[root@kudu1 ~]# scp -r kudu-1.15.0-install root@kudu3:/root

6. 启动(3台服务器)

6.1 启动master

因为启动一个master服务，需要连接其它的master服务，所以依次迅速的启动各个master

[root@kudu1 ~]# nohup kudu-1.15.0-install/usr/local/sbin/kudu-master --flagfile=kudu-1.15.0-install/usr/local/conf/master.gflagfile > kudu-1.15.0-install/usr/local/sbin/kudu-master.log 2>&1 &
[1] 9251
[root@kudu1 ~]#

分别访问http://kudu1:8051、http://kudu2:8051、http://kudu3:8051。其中http://kudu1:8051的页面如下所示：

6.2 启动tserver

[root@kudu1 ~]# nohup kudu-1.15.0-install/usr/local/sbin/kudu-tserver --flagfile=kudu-1.15.0-install/usr/local/conf/tserver.gflagfile > kudu-1.15.0-install/usr/local/sbin/kudu-tserver.log 2>&1 &
[2] 20995
[root@kudu1 ~]#

分别访问http://kudu1:8050、http://kudu2:8050、http://kudu3:8050。其中http://kudu1:8050的页面如下所示：

6.3 查看集群状态和数据完整性

ksck会发现问题。例如无法连接的tablet server、没有leader的tablet 、副本不足的tablet

查看集群状态命令

[root@kudu1 local]# pwd
/root/kudu-1.15.0-install/usr/local
[root@kudu1 local]# 
[root@kudu1 local]# bin/kudu cluster ksck kudu1:7051,kudu2:7051,kudu3:7051
Master Summary
               UUID               |     Address     | Status
----------------------------------+-----------------+---------
 62275082a3ab4be7a474c8ade4429616 |   kudu1:7051    | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b8 |   kudu2:7051    | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b9 |   kudu3:7051    | HEALTHY

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Tablet Server Summary
               UUID               |     Address     | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-----------------+---------+----------+----------------+-----------------
 7c6312b0200141d1ab7bde4990b2b577 |   kudu1:7050    | HEALTHY | <none>   |       0        |       0
 f21688849b7e4e1a901b2e580aba8e23 |   kudu2:7050    | HEALTHY | <none>   |       0        |       0
 f21688849b7e4e1a901b2e580aba8e24 |   kudu3:7050    | HEALTHY | <none>   |       0        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Version Summary
 Version |         Servers
---------+-------------------------
 1.15.0  | all 6 server(s) checked

The cluster doesn't have any matching tablets

The cluster doesn't have any matching system tables
The cluster doesn't have any matching tables
Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 0
 Third Quartile | 0
 Maximum        | 0


OK
[root@kudu1 local]# 

查看数据完整性命令
将一个tablet的所有副本数据进行计较，检验数据完整性

[root@kudu1 local]# 
[root@kudu1 local]# bin/kudu cluster ksck --checksum_scan --tables kudu_java_test --tablets 3fe1744a20e7433d864bb808a655c277,447b26717a33478abb4f004651e34aaf kudu1:7051,kudu2:7051,kudu3:7051
Checksum finished in 0s: 0/3 replicas remaining (0B from disk, 0 rows summed)
Master Summary
               UUID               |     Address     | Status
----------------------------------+-----------------+---------
 62275082a3ab4be7a474c8ade4429616 |    kudu1:7051   | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b8 |    kudu2:7051   | HEALTHY
 9ba2cdb6fc65440196af83d8ea3d52b8 |    kudu3:7051   | HEALTHY

Flags of checked categories for Master:
        Flag         |                            Value                            |         Master
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Tablet Server Summary
               UUID               |     Address     | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+-----------------+---------+----------+----------------+-----------------
 7c6312b0200141d1ab7bde4990b2b577 |    kudu1:7050   | HEALTHY | <none>   |       6        |       0
 f21688849b7e4e1a901b2e580aba8e23 |    kudu2:7050   | HEALTHY | <none>   |       6        |       0
 f21688849b7e4e1a901b2e580aba8e23 |    kudu3:7050   | HEALTHY | <none>   |       6        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       3

Flags of checked categories for Tablet Server:
        Flag         |                            Value                            |      Tablet Server
---------------------+-------------------------------------------------------------+-------------------------
 builtin_ntp_servers | 0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org | all 3 server(s) checked
 time_source         | system                                                      | all 3 server(s) checked

Version Summary
 Version |         Servers
---------+-------------------------
 1.15.0  | all 6 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
      Name      | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
----------------+----+---------+---------------+---------+------------+------------------+-------------
 kudu_java_test | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 0
 First Quartile | 0
 Median         | 3
 Third Quartile | 3
 Maximum        | 3

Checksum Summary
-----------------------
kudu_java_test
-----------------------
T 3fe1744a20e7433d864bb808a655c277 P f21688849b7e4e1a901b2e580aba8e23 (kudu3:7050): Checksum: 0
T 447b26717a33478abb4f004651e34aaf P f21688849b7e4e1a901b2e580aba8e23 (kudu3:7050): Checksum: 0

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 3
 Tablet Servers | 3
 Tables         | 1
 Tablets        | 3
 Replicas       | 3

OK
[root@kudu1 local]#