ambari 2.7.4搭建过程

最新推荐文章于 2024-04-23 11:46:27 发布

chefjuicy

最新推荐文章于 2024-04-23 11:46:27 发布

阅读量676

点赞数 1

文章标签： ambari

原文链接：https://blog.csdn.net/tang86100/article/details/108732111

版权

1、系统环境

搭建过程主要参考：https://blog.csdn.net/tang86100/article/details/108732111

系统：Centos7.8

配置：4台虚拟机，8核，32G，2T硬盘

备注：在搭建ambari2.7.4之前，尝试搭建ambari2.5.0，但是失败告终，找遍了所有方法都不行，个人感觉有可能centos版本太高，所以换了ambari2.7.4

集群规划

hadoop01 主节点 10.200.117.99

hadoop02 从节点 10.200.117.97

hadoop03 从节点 10.200.117.85

hadoop04 备用节点，用于后续扩容尝试 10.200.117.80

2、准备工作

2.1 关闭防火墙（每个节点都需要）

//暂停防火墙服务
systemctl stop firewalld.service

//设置防火墙服务开机不自动启动
systemctl disable firewalld.service

//查看防火墙服务开机不自启动设置成功，disabled表示设置成功
systemctl is-enabled firewalld.service

//查看防火墙状态
systemctl status firewalld.service

2.2 修改selinux（每个节点都需要）

//临时生效selinux设置
setenforce 0

//修改配置文件
vim /etc/selinux/config  #把selinux=enforcing修改为selinux=disabled

2.3 修改host（每个节点都一样，需要添加）

vi /etc/hosts

前两行不需要修改，直接在后面添加

10.200.117.99 hadoop01
10.200.117.97 hadoop02
10.200.117.85 hadoop03

2.4 修改network文件（每个节点都需要修改）

vim /etc/sysconfig/network

增加以下内容，如果hadoop01节点则HOSTNAME=hadoop01，hadoop02节点，则HOSTNAME=hadoop02，类推

#Created by anaconda
NETWORKING=yes
HOSTNAME=hadoop01

2.5 设置主机名（hadoop01节点设置主机名hadoop01，hadoop02节点设置主机名hadoop02，类推）

//设置主机名
hostnamectl set-hostname hadoop01

//查询主机名
hostname

2.6 修改yum源（所有节点）

//先安装wget
yum install -y wget

//备份yum源
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup

//下载阿里云repo文件
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

//生成缓存
yum makecache

2.7 安装ntp（所有节点）

// 安装 ntp 服务
yum install -y ntp
 
// 开启并设置开机自启ntp服务
systemctl start ntpd
systemctl enable ntpd
 
// 查看ntp服务状态
systemctl status ntpd

2.8.安装JDK（所有节点）

自行安装就行了，我的版本是jdk1.8.0_141

路径 /usr/java/jdk1.8.0_141

2.9.关闭THP（所有节点）

cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/enabled

得到的结果都是[always] madvise never，需要将其关闭，否则会占用很多资源

vim /etc/rc.d/rc.local

在文件末尾追加代码

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
        echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

并给该文件赋权

chmod +x /etc/rc.d/rc.local

reboot之后再执行查看命令，结果变为always madvise [never]，说明已关闭

2.10 打开最大文件设置（所有节点）

检查最大文件描述符数的当前值

ulimit -Sn
ulimit -Hn

//如果输出小于10000，则设置为10000

ulimit -n 10000

三、安装Ambari Server

3.1 开启免密登录（每台机器）

ssh-keygen -t rsa   //每个节点均需要，一直回车即可，如果已经生成过，可以overwrite

在~/.ssh下面会生成id_rsa和id_rsa.pub两个文件，再使用ssh-copy-id命令将公钥传递到各台机器上(包含自己）

ssh-copy-id -i .ssh/id_rsa.pub root@10.200.117.99
ssh-copy-id -i .ssh/id_rsa.pub root@10.200.117.97
ssh-copy-id -i .ssh/id_rsa.pub root@10.200.117.85

//每台机器都要重复这三行命令

此时就可以免密访问别的机器了

ssh root@10.200.117.97

3.2 安装MySQL数据库（我的规划是安装在hadoop01)

// 下载 Mysql RPM包
wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
 
// 安装MySQL RPM包
yum install -y mysql57-community-release-el7-10.noarch.rpm
 
// 安装MySQL和 Mysql-java连接器
yum -y install mysql-community-server mysql-connector-java
 
// 查看下载后的jar包，看目录中是否有mysql-connector-java:
ls /usr/share/java
 
// 启动Mysql服务
systemctl start mysqld.service
 
// 查看Mysql服务状态
systemctl status mysqld.service
 
// 设置Mysql服务开机自启
systemctl enable mysqld.service

接下来操作MySQL

//查看初始密码
cat /var/log/mysqld.log | grep password

//进入MySQL
mysql -u root -p

#输入密码

进入mysql后修改以下配置

//设置MySQL密码安全强度为0，最低长度为1
set global validate_password_policy=0;
set global validate_password_length=1;
 
// 修改root用户密码
ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
 
// 为root用户授予所有权限并设置本地登录
grant all privileges on *.* to 'root'@'localhost' identified by '123456';
 
// 为root用户授予所有权限并设置远程登录
grant all privileges on *.* to 'root'@'%' identified by '123456';
 
// 刷新权限系统
flush privileges;

由于安装了Yum Repository，每次yum操作都会自动更新，需要把这个卸载

yum -y remove mysql57-community-release-el7-10.noarch

再次进入mysql，创建ambari数据库及数据库的用户名和密码


create database ambari character set utf8;
 
CREATE USER 'ambari'@'%'IDENTIFIED BY '123456';
 
GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
 
create database hive character set utf8;
 
CREATE USER 'hive'@'%'IDENTIFIED BY '123456';
 
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
 
create database oozie character set utf8;
 
CREATE USER 'oozie'@'%'IDENTIFIED BY '123456';
 
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
 
CREATE DATABASE ranger;
 
CREATE USER 'rangerdba'@'localhost' IDENTIFIED BY 'rangerdba';
 
GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost';
 
CREATE USER 'rangerdba'@'%' IDENTIFIED BY 'rangerdba';
 
GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%';
 
GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'localhost' WITH GRANT OPTION;
 
GRANT ALL PRIVILEGES ON *.* TO 'rangerdba'@'%' WITH GRANT OPTION;
 
FLUSH PRIVILEGES;

3.3 安装nginx

yum install -y epel-release

yum install -y nginx

修改 /etc/nginx/nginx.conf文件

// 对文件进行备份
cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak
 
vi /etc/nginx/nginx.conf
 
//注释掉server中的root字段，在在location中添加以下内容
root ambari;
autoindex on;
autoindex_exact_size on;
autoindex_localtime on;

启动并设置开机自启Nginx服务

systemctl start nginx
systemctl enable nginx

下载软件包：（使用迅雷下载，好像还得充个会员，速度才快，其他下载软件不咋行）

一般来说我们都是在windows上操作集群

我自己的步骤：在windows上用迅雷下载以下安装包→从windows上传安装包到hadoop01机器（因为hdp比较大，我用filezilla，支持断点续传），上传到/usr/soft

Ambari 2.7.4：http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz

HDP：http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz
HDP-UTILS：http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
HDP-GPL：http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz

创建目录，将上传的软件包解压到这里

mkdir /usr/share/nginx/ambari

tar -zxvf HDP-GPL-3.1.4.0-centos7-gpl.tar.gz -C /usr/share/nginx/ambari/
tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz -C /usr/share/nginx/ambari/
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /usr/share/nginx/ambari/
tar -zxvf ambari-2.7.4.0-centos7.tar.gz -C /usr/share/nginx/ambari/

3.4 安装Ambari-server（hadoop01节点）

下载相关yum工具

yum install -y yun-utils
yum repolist
yum install -y createrepo

配置Ambari Repo

//找到以下路径，修改ambari.repo
cd /usr/share/nginx/ambari/ambari/centos7/2.7.4.0-118
 
vi ambari.repo
 
//修改为以下内容
#VERSION_NUMBER=2.7.4.0-118
[ambari-2.7.4.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.4.0
baseurl=http://192.168.199.123/ambari/centos7/2.7.4.0-118/
gpgcheck=1
gpgkey=http://192.168.199.123/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
 
//将文件拷贝到以下地址
cp ambari.repo /etc/yum.repos.d/
 
//修改HDP.repo
cd /usr/share/nginx/ambari/HDP/centos7/3.1.4.0-315
vi hdp.repo
 
//添加以下内容
#VERSION_NUMBER=3.1.4.0-315
[HDP-3.1.4.0-315]
name=HDP Version - HDP-3.1.4.0-315
baseurl=http://192.168.199.123/HDP/centos7
gpgcheck=1
gpgkey=http://192.168.199.123/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
 
[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.199.123/HDP-UTILS/
gpgcheck=1
gpgkey=http://192.168.199.123/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
 
//将文件拷贝到以下地址
cp hdp.repo /etc/yum.repos.d/

将Repo配置文件分发到各个节点（这样其他机器才能访问源，下载文件）

scp /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/HDP.repo hadoop02:/etc/yum.repos.d/
scp /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/HDP.repo hadoop03:/etc/yum.repos.d/

清理以下yum源缓存

yum clean all

yum makecache

yum repolist

安装ambari-server（hadoop01）

yum install -y ambari-server

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

//一般其他教程都没有在ambari-server setup的时候声明jdbc driver，这样有可能会导致后期test 数据库连接的时候无法连通

按照以下内容选择

[root@hadoop01 centos7]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/java/jdk1.8.0_221
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): hadoop-01
Port (3306):
Database name (ambari):
Username (ambari): 123456
Invalid characters in username. Start with _ or alpha followed by alphanumeric or _ or - characters
Username (ambari):
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.4.0.118.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

使用ambari用户登录ambari数据库，导入默认数据

mysql -u ambari -p

use ambari

source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
 
show tables;

启动ambari-server


//启动ambari-server，一次不成再restart一次
ambari-server start
 
//ambari-server的其他操作命令
ambari-server restart
ambari-server stop
ambari-server reset
ambari-server setup

看到以下内容说明启动成功

Using python  /usr/bin/python
Restarting ambari-server
Waiting for server stop...
Ambari Server stopped
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...................................
Server started listening on 8080
 
DB configs consistency check: no errors and warnings were found.

访问Ambari Web页面，默认端口8080，Username：admin；Password：admin

http://10.200.117.99:8080

四、安装Ambari Agent

分别在各节点执行以下命令

yum install ambari-agent -y

整个Ambari平台安装完成

五、安装部署HDP集群

1、登录安装向导（这里采用了原文的截图）

2、给你的集群取名字

3.选择HDP版本，修改本地源地址

把IP地址替换为hadoop01的ip地址即可

4、添加主机，上传私钥，就是~/.ssh/id_rsa (hadoop01的，先把它传到windows再上传）

对应我的Target Hosts就是：hadoop01 hadoop02 hadoop03

5.大数据组件安装，需要哪些勾选哪些（我没有选择Pig、accumulo、Druid、storm、ranger)

6.节点配置

7、分配节点

8.配置安全密码（Hive以及oozie均设置为123456）

9.数据路径配置（不懂的可以不用改）

10.用户配置

11.全部配置

12.安装预览

13.接下来就是漫长的等待

14.最终安装成功

备注：最开始我是有选择安装ranger，但是最终安装失败，所以去掉了ranger，打算后期手动安装。集群部署好核心组件即可，其他辅助组件与特色组件可以手动安装

15.部署HDP成功后，启动各Services组件，消灭左边的所有报红项，平台就可使正常使用了。

chefjuicy

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
2
评论
ambari 2.7.4搭建过程

1、系统环境搭建过程主要参考：https://blog.csdn.net/tang86100/article/details/108732111系统：Centos7.8配置：4台虚拟机，8核，32G，2T硬盘备注：在搭建ambari2.7.4之前，尝试搭建ambari2.5.0，但是失败告终，找遍了所有方法都不行，个人感觉有可能centos版本太高，所以换了ambari2.7.4集群规划hadoop01 主节点hadoop02 从节点hadoop03 从节点hado.
复制链接

扫一扫