1.安装前准备
1.1配置jdk环境变量
# jdk setting
export JAVA_HOME=/data/base/jdk1.8.0_211
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
1.2 yum更换阿里云源
# curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
#清理缓存
# yum clean all
#生成新的缓存
# yum makecache
1.3配置时钟同步
yum -y install ntpdate
ntpdate -u ntp.aliyun.com
# 配置定时任务
vi /etc/crontab
*/10 * * * * /usr/sbin/ntpdate ntp.aliyun.com
1.4 关闭selinux
selinux的主要作用就是最大限度地减小系统中服务进程可访问的资源(最小权限原则)
vim /etc/sysconfig/selinux
# SELINUX=enforcing
SELINUX=disabled
1.5 关闭 THP (Transparent Huge Page)
THP的目的是通过将页表项映射更大的内存,来减少 Page Fault,从而提升 TLB (Translation Lookaside Buffer,由存储器管理单元用于改进虚拟地址到物理地址的转译速度)的命中率。结合存储器层次结构设计原理可知,当程序的访存局部性较好时,THP 将带来性能提升,反之 THP 的优势不仅丧失,还有可能化身为恶魔,引起系统的不稳定。查看THP状态
[root@bdp-ambari base]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
[root@bdp-ambari base]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
如果值是always,则修改/etc/rc.d/rc.local 关闭THP
touch /var/lock/subsys/local
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
或者直接执行以下操作关闭THP
[root@bdp-ambari base]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@bdp-ambari base]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
1.6 修改文件句柄最大限制(需要重启机器)
vim /etc/security/limits.conf
# End of file
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
1.7 配置SSH无密码登录
ssh-keygen -t rsa # 生成公钥
#将生成的公钥添加到免密登陆keys中(主节点)
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
1.8 安装mysql和数据库创建用户
Mysql系列2:Mysql5.7.42 一键离线安装脚本_shangjg3的博客-CSDN博客
CREATE DATABASE ambari;
use ambari;
CREATE USER 'ambari'@'%' IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
FLUSH PRIVILEGES;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
show tables;
use mysql;
select Host,User from user where user='ambari';
CREATE DATABASE hive;
use hive;
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
FLUSH PRIVILEGES;
CREATE DATABASE oozie;
use oozie;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';
FLUSH PRIVILEGES;
1.9 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
2.修改yum源,实现离线安装
2.1安装httpd
[root@bdp-ambari data]# yum -y install httpd
[root@bdp-ambari data]# service httpd restart
Redirecting to /bin/systemctl restart httpd.service
[root@bdp-ambari data]# chkconfig httpd on
2.2 部署ambari文件目录
安装完成后会生成/var/www/html 目录,在此目录下创建ambari目录,并将以下三个文件上传到ambari目录
cd /var/www/html
mkdir ambari
ambari-2.7.4.0-centos7.tar.gz
HDP-3.1.4.0-centos7-rpm.tar.gz
HDP-UTILS-1.1.0.22-centos7.tar.gz
解压文件
mkdir HDP-UTILS
tar -zxvf ambari-2.7.4.0-centos7.tar.gz
tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C HDP-UTILS
rm -f *.gz
浏览器输入地址:http://10.16.60.81/ambari/
2.3 制作本地yum源
vim ambari/centos7/2.7.4.0-118/ambari.repo
#VERSION_NUMBER=2.7.4.0
[ambari-2.7.4.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.4.0
baseurl=http://10.16.60.81/ambari/ambari/centos7/2.7.4.0-118
gpgcheck=1
gpgkey=http://10.16.60.81/ambari/ambari/centos7/2.7.4.0-118/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
cp ambari/centos7/2.7.4.0-118/ambari.repo /etc/yum.repos.d/
vim /var/www/html/ambari/HDP/centos7/3.1.4.0-315/hdp.repo
[HDP-3.1]
name=HDP Version - HDP-3.1
baseurl=http://10.16.60.81/ambari/HDP/centos7/3.1.4.0-315
gpgcheck=1
gpgkey=http://10.16.60.81/ambari/HDP/centos7/3.1.4.0-315/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1]
name=HDP-UTILS Version - HDP-UTILS-1.1
baseurl=http://10.16.60.81/ambari/HDP-UTILS/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://10.16.60.81/ambari/HDP-UTILS/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
cp HDP/centos7/3.1.4.0-315/hdp.repo /etc/yum.repos.d/
创建ambari和hdp的yum元数据缓存
yum clean all
yum makecache
yum repolist
/usr/bin/yum list available --showduplicates --disablerepo=* --enablerepo=HDP-3.1
3.安装ambari-server
3.1安装ambari-server
yum -y install ambari-server
3.2配置ambari-server
ambari-server setup
[root@bdp-ambari ambari]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /data/base/jdk1.8.0_211
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? n
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK
About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.7.4.0.118.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
使用mysql数据库
[root@amaster resources]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n^H^H^H
input not recognized, please try again:
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):ambari
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /data/base/jdk1.8.0_211
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.4.0.118.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
3.3启动ambari-server
[root@bdp-ambari ambari]# ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start........................
Server started listening on 8080
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
浏览器输入访问地址:http://10.16.60.81:8080/#/login, 可以看到ambari的web登录页面。
输入默认用户名密码:admin/admin登录。
4.安装HDP
4.1选择HDP版本和配置baseUrl
http://10.16.60.81/ambari/HDP/centos7/3.1.4.0-315
http://10.16.60.81/ambari/HDP-UTILS/HDP-UTILS/centos7/1.1.0.22
4.2配置主机
4.3注册主机
4.4选择要安装的服务
4.5分配大数据组件的master服务到集群内的各个主机
4.6分配大数据组件的slave服务或者client到集群内的各个主机
4.7配置各个组件的密码
4.8 配置各个大数据组件的详细信息
配置druid、hvie、oozie、ranger、rangerkms和superset的元数据存储信息,这里使用mysql数据库。
设置mysql driver
[root@bdp data]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/data/base/mysql-conn/mysql-connector-java-5.1.47.jar
Using python /usr/bin/python
Setup ambari-server
Copying /data/base/mysql-conn/mysql-connector-java-5.1.47.jar to /var/lib/ambari-server/resources/mysql-connector-java-5.1.47.jar
Creating symlink /var/lib/ambari-server/resources/mysql-connector-java-5.1.47.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java-5.1.47.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.