Ambari2.7.3+HDP3.0.1安装HBase+kerberos

最新推荐文章于 2023-11-27 08:39:31 发布

Jiang_James_L

最新推荐文章于 2023-11-27 08:39:31 发布

阅读量2.6k

点赞数

分类专栏： Linux

本文链接：https://blog.csdn.net/u014305062/article/details/89303667

版权

Linux 专栏收录该内容

6 篇文章 0 订阅

订阅专栏

1 安装前准备

1.1 主机列表

本次实验选择6台主机，1台作为Ambari Server主机，5台作为Ambari Agent

节点ip	OS版本	jdk版本	hostname -f	hostname	安装软件
192.168.137.129	CentOS7.4	openjdk1.8.0_131	server.ambari	ambari137129	Ambari Server
192.168.137.130	CentOS7.4	openjdk1.8.0_131	slave1.ambari	ambari137130	Ambari Agent
192.168.137.131	CentOS7.4	openjdk1.8.0_131	slave2.ambari	ambari137131	Ambari Agent
192.168.137.132	CentOS7.4	openjdk1.8.0_131	slave3.ambari	ambari137132	Ambari Agent
192.168.137.133	CentOS7.4	openjdk1.8.0_131	slave4.ambari	ambari137133	Ambari Agent
192.168.137.134	CentOS7.4	openjdk1.8.0_131	slave5.ambari	ambari137134	Ambari Agent

注意：

Ambari Server主机到Ambari Agent主机免密登录
Ambari Server/Agent主机需安装JDK
确保主机的hostname -f 满足FQDN格式（在安装集群的第三步Confirm Host需要）
关闭防火墙
确认主机字符集编码为UTF-8（否则Ambari Server 配置数据库可能报错）
开启NTP服务

1.2 安装基本软件包

在所有的主机上安装

yum install openssh wget vim openssh-clients openssl openssh-server -y
systemctl enable sshd && systemctl start sshd

1.3 配置本地yum源

1.3.1 下载安装介质

Ambari2.7.3：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari-2.7.3.0-centos7.tar.gz
HDP3.0.1：http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.1.0/HDP-3.0.1.0-centos7-rpm.tar.gz
HDP-UTILS1.1.0.22：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

注意：

到hortonworks官网查看HDP对应的Hadoop和Hbase版本，选择正确的版本进行下载
安装介质较大

1.3.2 配置HTTP yum源

因Ambari安装集群界面不能识别ftp地址，因此配置成http yum源。在192.168.137.129上安装httpd服务：

yum -y install httpd
systemctl start httpd
systemctl enable httpd

将上一步下载的压缩包解压之后，放到/var/www/html下。通过浏览器访问http://192.168.137.129:18080/ambari2.7.3/centos7/测试http yum源是否正常。

修改解压目录下里的ambari.repo、hdp.repo baseurl地址为http yum源地址。

ambari.repo

#VERSION_NUMBER=2.7.3.0-139
[ambari-2.7.3.0]
name=ambari Version - ambari-2.7.3.0
baseurl=http://192.168.137.129:18080/ambari2.7.3/centos7/2.7.3.0-139/
gpgcheck=1
gpgkey=http://192.168.137.129:18080/ambari2.7.3/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

hdp.repo

#VERSION_NUMBER=3.0.1.0-187
[HDP-3.0.1.0]
name=HDP Version - HDP-3.0.1.0
baseurl=http://192.168.137.129:18080/HDP-3.0.1.0/centos7/3.0.1.0-187/
gpgcheck=1
gpgkey=http://192.168.137.129:18080/HDP-3.0.1.0/centos7/3.0.1.0-187/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.137.129:18080/HDP-UTILS-1.1.0.22/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://192.168.137.129:18080/HDP-UTILS-1.1.0.22/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

将ambari.repo、hdp.repo拷贝至192.168.137.129的/etc/yum.repos.d目录下。

2 安装Ambari Server

2.1 以默认方式安装Ambari Server

下载postgresql-42.2.5.jar，放置在192.168.137.129:/opt。
在192.168.137.129主机执行命令：

yum install -y ambari-server
ambari-server setup --jdbc-db=postgres --jdbc-driver=/opt/postgresql-42.2.5.jar
ambari-server setup

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/lib/jvm/java-openjdk
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? Y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 1         ####如果主机字符集编码未设置正确，可能会启动报错，具体可以查看日志/var/log/ambari-server/ambari-server.log
Database admin user (postgres):
Database name (ambari):
Postgres schema (ambari):
Username (ambari):
Enter Database Password (bigdata):
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

ambari-server start

通过浏览器访问http://192.168.137.129:8080测试是否正常启动，默认用户名密码为admin/admin。

3 部署HDP集群

3.1 登录

通过浏览器访问http://192.168.137.129:8080，默认用户名密码为admin/admin。

3.2 安装向导

点击"Launch Install Wizard"。

3.2.1 设置集群名字

设置集群名字：hbase_test

3.2.2 选择版本并配置为本地源

选择HDP版本 --> ADD New Version：http://192.168.137.129:18080/HDP-3.0.1.0/centos7/3.0.1.0-187/HDP-3.0.1.0-187.xml
选择本地源 --> HDP-3.0.1.0：http://192.168.137.129:18080/HDP-3.0.1.0/centos7/3.0.1.0-187/
HDP-UTILS-1.1.0.22：http://192.168.137.129:18080/HDP-UTILS-1.1.0.22/centos7/1.1.0.22/

3.2.3 安装配置

Targets Host

slave1.ambari
slave2.ambari
slave3.ambari
slave4.ambari
slave5.ambari

SSH Private Key

注意：

私钥为Ambari Server的root用户下的私钥

3.2.4 确认 Hosts

等待ambari-agent安装完成

注意：

失败可以点击Failed进行查看
ambari注册主机的时候，ambari-agent出现如下错误：
NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
解决方案在ambari-agent的配置文件ambari-agent.ini 在［security］增加一项
/etc/ambari-agent/conf/ambari-agent.ini
[security]
force_https_protocol=PROTOCOL_TLSv1_2

3.2.5 选择安装服务

安装服务列表

1 HDFS
2 YARN + MapReduce2
3 HBase
4 Zookeeper
5 Ambari Metrics

3.2.6 分配Master

尽量均匀分散。

3.2.7 分配Agent

Agent配置

DataNode ---- all
NodeManager ---- all
RegionServer ---- all
Client ---- all

3.2.8 安装、启动、测试服务

配置好服务之后，等待Ambari安装HDFS、HBase集群，安装完成后，Ambari会自动启动服务和测试服务，测试成功后，即可完成集群的安装。

3.2.9 集群状态

通过浏览器访问http://192.168.137.129:8080查看集群状态

4 部署Kerberos

4.1 安装软件包

ambari server

yum install -y krb5-server krb5-workstation

ambari client

yum install -y krb5-workstation

4.2 配置Kerberos

ambari server + agent 主机

vim /etc/krb5.conf

 default_realm = HBASE.TEST
 #default_ccache_name = KEYRING:persistent:%{uid}
 HBASE.TEST = {
  kdc = server.ambari
  admin_server = server.ambari
 }

注意：

default_realm自定义访问域名，建议大写
default_ccache_name建议注释
kdc\admin_server配置为ambari server主机的hostname -f

ambari server主机

vim /var/kerberos/krb5kdc/kdc.conf

 HBASE.TEST = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }

vim /var/kerberos/krb5kdc/kadm5.acl

*/admin@HBASE.TEST      *

4.3 初始化Kerberos database

ambari server主机

kdb5_util create -s -r DEV.HBASE.TEST
> KDC database master key: VnRVUxv5

4.4 添加database administrator

ambari server主机

kadmin.local
> addprinc admin/admin
> Enter password for principal "admin/admin@HBASE.TEST": admin1234

4.5 启动Kerberos

ambari server主机

systemctl start krb5kdc
systemctl enable krb5kdc
systemctl start kadmin
systemctl enable kadmin

4.6 登录测试

ambari agent主机

kadmin -p admin/admin@HBASE.TEST

4.7 登录 ambari 添加server节点

HOST --> Add New Host
添加节点

4.8 登录 ambari enable kerberos

Kerberos --> ENABLE KERBEROS

注意：

Realm name自定义访问域名，建议大写，需要与/etc/krb5.conf的default_realm匹配
KDC hosts\Kadmin host配置为ambari server主机的hostname -f
Admin principal配置为4.4步创建的principaladmin/admin
Admin password配置为4.4步创建的密码admin1234