在已有的Hadoop集群中安装Kudu

最新推荐文章于 2023-08-29 15:54:27 发布

汐朔

最新推荐文章于 2023-08-29 15:54:27 发布

阅读量936

点赞数 1

分类专栏： Kudu hadoop 文章标签： Kudu 安装kudu kudu安装 hadoop安装kudu hadoop

本文链接：https://blog.csdn.net/arlanhon/article/details/98852372

版权

hadoop 同时被 2 个专栏收录

21 篇文章 0 订阅

订阅专栏

Kudu

6 篇文章 0 订阅

订阅专栏

背景

1、网上找了很多资料，所有的安装Kudu的教程都是从cloudera-kudu（即CDH-kudu），而我们公司及身边的朋友基本也没用CDH版的，而且是用的apache hadoop，是否会有兼容性问题？这也是写本篇文章的原由之一，以此来验证。
2、本来只是想了解下kudu的api, 没想到道路还这么波折。相比之下比安装hadoop， spark要复杂很多。

安装前准备

查看集群环境并确定服务安装位置

hadoop version

[daxiang@isunode011 yum.repos.d]$ hadoop version
Hadoop 2.7.3.2.6.0.3-8
Subversion git@github.com:hortonworks/hadoop.git -r c6befa0f1e911140cc815e0bab744a6517abddae
Compiled by jenkins on 2017-04-01T21:32Z
Compiled with protoc 2.5.0
From source with checksum a28afaaa6b3713cb3163c88551b1c48
This command was run using /usr/hdp/2.6.0.3-8/hadoop/hadoop-common-2.7.3.2.6.0.3-8.jar

more /etc/hosts
在这里插入图片描述
因为只是测试，所以只选用三台服务器安装。
配置如下：

服务	IP	HOST名
kudu-master	192.168. 220.145	isunode011
kudu-tserver	192.168. 220.146	isunode012
kudu-tserver	192.168. 220.147	isunode013

查看centos系统版本

执行命令cat /etc/issue，确定好系统版本为centos6

[daxiang@isunode011 ~]$ cat /etc/issue
CentOS release 6.6 (Final)
Kernel \r on an \m

执行命令more /etc/fstab, 按照Kudu官网的安装要求，磁盘的格式要为ext4

[daxiang@isunode011 ~]$ more /etc/fstab
#
# /etc/fstab
# Created by anaconda on Tue Feb 21 14:26:58 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=44721b03-dee7-4208-8b4d-2f76cfa64e65 /                       ext4    defaults        1 1
UUID=e6684031-4053-46d9-afeb-4c01d8e6f48b /boot                   ext4    defaults        1 2
UUID=a09ea52e-3723-488d-b47f-cde570baefae /home                   ext4    defaults        1 2
UUID=205432de-8fa9-432c-be96-58ac1875e7e2 swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=data1             /data1_4T               ext4    defaults        0 0
LABEL=data2             /data2_4T               ext4    defaults        0 0

安装依赖包

yum -y install cyrus-sasl-plain lsb ntp

配置cloudera-kudu.repo源

上面确定好为centos6版本后，进入执行命令wget http://archive.cloudera.com/kudu/redhat/6/x86_64/kudu/cloudera-kudu.repo，
如果是centos7，则执行命令wget http://archive.cloudera.com/kudu/redhat/7/x86_64/kudu/cloudera-kudu.repo
cd到yum.repos.d目录cd /etc/yum.repos.d
然后执行如下命令即可。

[daxiang@isunode011 yum.repos.d]$ cd /etc/yum.repos.d/
[daxiang@isunode011 yum.repos.d]$ wget http://archive.cloudera.com/kudu/redhat/6/x86_64/kudu/cloudera-kudu.repo
--2019-08-08 11:31:27--  http://archive.cloudera.com/kudu/redhat/6/x86_64/kudu/cloudera-kudu.repo
Resolving archive.cloudera.com... 151.101.228.167
Connecting to archive.cloudera.com|151.101.228.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 321 [binary/octet-stream]
cloudera-kudu.repo: Permission denied

Cannot write to “cloudera-kudu.repo” (Permission denied).
[daxiang@isunode011 yum.repos.d]$ sudo wget http://archive.cloudera.com/kudu/redhat/6/x86_64/kudu/cloudera-kudu.repo
--2019-08-08 11:31:34--  http://archive.cloudera.com/kudu/redhat/6/x86_64/kudu/cloudera-kudu.repo
Resolving archive.cloudera.com... 151.101.228.167
Connecting to archive.cloudera.com|151.101.228.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 321 [binary/octet-stream]
Saving to: “cloudera-kudu.repo”

100%[====================================================================================================================================================================================================================================>] 321         --.-K/s   in 0s      

2019-08-08 11:31:35 (52.3 MB/s) - “cloudera-kudu.repo” saved [321/321]

安装kudu-master

yum install kudu kudu-master kudu-client0 kudu-client-devel -y
在这里插入图片描述
发现下载速度太慢了，将CentOS-Base.repo源替换成阿里的。
分别执行
mv CentOS-Base.repo CentOS-Base.repo.backup20190808
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo

[root@isunode011 yum.repos.d]# mv CentOS-Base.repo CentOS-Base.repo.backup20190808
[root@isunode011 yum.repos.d]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo
--2019-08-08 11:48:00--  http://mirrors.aliyun.com/repo/Centos-6.repo
Resolving mirrors.aliyun.com... 222.188.8.228, 222.188.8.230, 222.188.8.225, ...
Connecting to mirrors.aliyun.com|222.188.8.228|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2523 (2.5K) [application/octet-stream]
Saving to: “/etc/yum.repos.d/CentOS-Base.repo”

100%[====================================================================================================================================================================================================================================>] 2,523       --.-K/s   in 0s      

2019-08-08 11:48:00 (384 MB/s) - “/etc/yum.repos.d/CentOS-Base.repo” saved [2523/2523]

替换后速度杠杠的
在这里插入图片描述

修改配置

sudo vim /etc/kudu/conf/master.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-master.
--fromenv=rpc_bind_addresses
--fromenv=log_dir

# 默认位置
#--fs_wal_dir=/var/lib/kudu/master
#--fs_data_dirs=/var/lib/kudu/master

# 修改后的位置
--fs_wal_dir=/opt/kudu/master
--fs_data_dirs=/opt/kudu/master

mkdir /opt/kudu && chown kudu:kudu /opt/kudu (此命令要root帐户执行)

启动

执行/etc/init.d/kudu-master start

[root@isunode011 opt]# /etc/init.d/kudu-master start
Failed to start Kudu Master Server. Return value: 1        [FAILED]

发现启动报错了，查看日志，
cd /var/log/kudu
tail -100f kudu-master.isunode011.kudu.log.FATAL.20190808-122227.23568

[root@isunode011 opt]# cd /var/log/kudu/
[root@isunode011 kudu]# ll
total 44
lrwxrwxrwx 1 kudu kudu   59 Aug  8 12:22 kudu-master.ERROR -> kudu-master.isunode011.kudu.log.ERROR.20190808-122227.23568
lrwxrwxrwx 1 kudu kudu   59 Aug  8 12:22 kudu-master.FATAL -> kudu-master.isunode011.kudu.log.FATAL.20190808-122227.23568
lrwxrwxrwx 1 kudu kudu   58 Aug  8 12:22 kudu-master.INFO -> kudu-master.isunode011.kudu.log.INFO.20190808-122227.23568
-rw------- 1 kudu kudu  320 Aug  8 12:00 kudu-master.isunode011.kudu.log.ERROR.20190808-120016.22708
-rw------- 1 kudu kudu  320 Aug  8 12:22 kudu-master.isunode011.kudu.log.ERROR.20190808-122227.23568
-rw------- 1 kudu kudu  320 Aug  8 12:00 kudu-master.isunode011.kudu.log.FATAL.20190808-120016.22708
-rw------- 1 kudu kudu  320 Aug  8 12:22 kudu-master.isunode011.kudu.log.FATAL.20190808-122227.23568
-rw------- 1 kudu kudu 1033 Aug  8 12:00 kudu-master.isunode011.kudu.log.INFO.20190808-120016.22708
-rw------- 1 kudu kudu 1033 Aug  8 12:22 kudu-master.isunode011.kudu.log.INFO.20190808-122227.23568
-rw------- 1 kudu kudu  320 Aug  8 12:00 kudu-master.isunode011.kudu.log.WARNING.20190808-120016.22708
-rw------- 1 kudu kudu  320 Aug  8 12:22 kudu-master.isunode011.kudu.log.WARNING.20190808-122227.23568
-rw-r--r-- 1 kudu kudu 1278 Aug  8 12:22 kudu-master.out
lrwxrwxrwx 1 kudu kudu   61 Aug  8 12:22 kudu-master.WARNING -> kudu-master.isunode011.kudu.log.WARNING.20190808-122227.23568
drwx------ 3 kudu kudu 4096 Aug  8 12:00 minidumps
[root@isunode011 kudu]# tail -100f kudu-master.isunode011.kudu.log.FATAL.20190808-122227.23568
Log file created at: 2019/08/08 12:22:27
Running on machine: isunode011
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0808 12:22:27.793143 23568 master_main.cc:68] Check failed: _s.ok() Bad status: Service unavailable: Cannot initialize clock: Error reading clock. Clock considered unsynchronized

解决异常
安装ntp服务：
yum install ntp
/etc/init.d/ntpd restart

[root@isunode011 kudu]# yum install ntp
Loaded plugins: fastestmirror, security
Setting up Install Process
Repository HDP-2.6.0.3 is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Package ntp-4.2.6p5-15.el6.centos.x86_64 already installed and latest version
Nothing to do
[root@isunode011 kudu]# /etc/init.d/ntpd restart
Shutting down ntpd:                                        [FAILED]
Starting ntpd:                                             [  OK  ]

重启kudu-master后，成功了。如下:

[root@isunode011 kudu]# /etc/init.d/kudu-master start
Started Kudu Master Server (kudu-master):                  [  OK  ]

安装kudu-tserver

因为其它几台机器都没权限操作，所以这里只进行单机安装。如果是多机进行安装，操作是一样。只不过在启动之前要执行ntp同步ntpdate us.pool.ntp.org。
yum install kudu kudu-tserver kudu-client0 kudu-client-devel -y
在这里插入图片描述

修改配置

vim /etc/kudu/conf/tserver.gflagfile

# Do not modify these two lines. If you wish to change these variables,
# modify them in /etc/default/kudu-tserver.
--fromenv=rpc_bind_addresses
--fromenv=log_dir

#--fs_wal_dir=/var/lib/kudu/tserver
#--fs_data_dirs=/var/lib/kudu/tserver

--fs_wal_dir=/opt/kudu/tserver
--fs_data_dirs=/opt/kudu/tserver
--tserver_master_addrs=isunode011:7051

启动

/etc/init.d/kudu-tserver start

[root@isunode011 kudu]# /etc/init.d/kudu-tserver start
Started Kudu Tablet Server (kudu-tserver):                 [  OK  ]

界面

安装并启动成功后，如下界面
在这里插入图片描述

结论

kudu的安装与是不是CDH版的跟已有的hadoop集群不冲突。

汐朔

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
在已有的Hadoop集群中安装Kudu

目录背景安装前准备查看集群环境并确定服务安装位置查看centos系统版本安装依赖包配置cloudera-kudu.repo源安装kudu-master修改配置启动安装kudu-tserver修改配置启动界面结论背景1、网上找了很多资料，所有的安装Kudu的教程都是从cloudera-kudu（即CDH-kudu），而我们公司及身边的朋友基本也没用CDH版的，而且是用的apache hadoop...
复制链接

扫一扫