centos 7 环境使用ambari离线搭建hadoop集群

hadoop介绍

https://baike.baidu.com/item/Hadoop/3526507

http://hadoop.apache.org/

ambari介绍

https://baike.baidu.com/item/Ambari/19697889

http://ambari.apache.org/
 

基础环境

1.服务器

ip	hostname	resouces
172.16.100.10	node-1	2C+8G+30G
172.16.100.11	node-2	2C+8G+30G
172.16.100.12	node-3	2C+8G+30G
172.16.100.13	node-4	2C+8G+30G
172.16.100.14	node-5	2C+8G+30G

2.software

soft	version
centos	CentOS Linux release 7.5
mariadb	5.5.56
jdk	1.8.0_112
ambari	2.6.1.0
HDP	2.6.4.0
HDP-GPL	2.6.4.0
HDP-UTILS	1.1.0.22

3.基础环境配置

• 3.1关闭防火墙

[root@node-1 conf]# systemctl stop firewalld.service

[root@node-1 conf]# systemctl disable firewalld.service

• .3.2关闭NetworkManager

[root@node-1 /]# systemctl stop NetworkManager

[root@node-1 /]# systemctl disable NetworkManager

• 3.3关闭selinux，vim /etc/selinux/config，重启生效。

SELINUX=disabled

• 3.4ssh 免密登陆

      创建ssh免密登陆脚本

[root@node-1 ~]# vim configure_ssh_without_pass

#!/bin/bash


ssh-keygen -t rsa -b 2048 -N "" -f $HOME/.ssh/id_rsa

cat $HOME/.ssh/id_rsa.pub >$HOME/.ssh/authorized_keys
chmod 600 $HOME/.ssh/authorized_keys

for ip in $(awk '{print $1}' ip.config); do
    #rsync -av -e 'ssh -o StrictHostKeyChecking=no' $HOME/.ssh/authorized_keys root@$ip:$HOME/.ssh/
    scp -p -o StrictHostKeyChecking=no $HOME/.ssh/authorized_keys root@$ip:$HOME/.ssh/
done

[root@node-1 ~]# vim ip.config

172.16.100.10 node-1
172.16.100.11 node-2
172.16.100.12 node-3
172.16.100.13 node-4
172.16.100.14 node-5

  根据实际情况修改ip.config服务器ip信息，运行configure_ssh_without_pass.sh脚本，根据提示完成操作，并完成ssh免密登陆验证。

[root@node-1 ~]# bash configure_ssh_without_pass 

• 3.5集群节点开启ntp服务

  安装ntp，同步公网时间同步服务器，并添加定时任务。

[root@node-1 ~]# yum install ntpdate ntp -y

[root@cluster-slave4 /]# ntpdate -u 1.cn.pool.ntp.org

[root@node-1 ~]# crontab  -e
0 0 * * * /usr/sbin/ntpdate -u 1.cn.pool.ntp.org;hwclock -w

•  3.6集群节点配置hosts

[root@node-1 ~]# cat ip.config >>/etc/hosts

4.配置本地yum源（172.16.100.10）

• 4.1安装httpd

[root@node-1 ~]#  yum install httpd -y
[root@node-1 ~]# systemctl start httpd
[root@node-1 ~]# systemctl status httpd

• 4.2下载官方镜像文件，其中包含两个repo文件和4个压缩包：

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari.repo
wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/hdp.repo
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari-2.6.1.0-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.4.0/HDP-GPL-2.6.4.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz

• 4.3将对应的tar包解压到httpd的html文件目录

tar -C /var/www/html/ -zxf ambari-2.6.1.0-centos7.tar.gz
tar -C /var/www/html/ -zxf HDP-2.6.4.0-centos7-rpm.tar.gz
tar -C /var/www/html/ -zxf HDP-GPL-2.6.4.0-centos7-rpm.tar.gz
tar -C /var/www/html/ -zxf HDP-UTILS-1.1.0.22-centos7.tar.gz

• 4.4配置ambari、HDP、HDP-UTILS的本地源

vim ambari.repo

#VERSION_NUMBER=2.6.1.0-143
# ambari 源

[ambari-2.6.1.0]
name=ambari Version - ambari-2.6.1.0
baseurl=http://172.16.100.10/ambari/centos7/2.6.1.0-143
gpgcheck=1
gpgkey=http://172.16.100.10/ambari/centos7/2.6.1.0-143/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

vim hdp.repo
#VERSION_NUMBER=2.6.4.0-91

[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://172.16.100.10/HDP/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://172.16.100.10/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://172.16.100.10/HDP-UTILS/
gpgcheck=1
gpgkey=http://172.16.100.10/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-GPL-2.6.4.0]
name=HDP-GPL Version - HDP-GPL-2.6.4.0
baseurl=http://172.16.100.10/HDP-GPL/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://172.16.100.10/HDP-GPL/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

• 4.5将上面的修改过的源放到/etc/yum.repos.d/下面。

cp ambari.repo /etc/yum.repos.d/
cp hdp.repo /etc/yum.repos.d/

• 4.6更新yum

yum clean all
yum list update
yum makecache
yum repolist

5.安装配置数据库

[root@node-1 ~]# yum install mariadb-server -y
[root@node-1 ~]# systemctl start mariadb
[root@node-1 ~]# mysql_secure_installation

•  5.1 创建业务数据库

MariaDB [(none)]> create database ambari default character set utf8;
MariaDB [(none)]> grant all on ambari.* to ambari@localhost identified by 'bigdata';
MariaDB [(none)]> grant all on ambari.* to ambari@'%' identified by 'bigdata';
MariaDB [(none)]> create database hive default character set utf8;
MariaDB [(none)]> grant all on hive.* to hive@localhost identified by 'hive';
MariaDB [(none)]> grant all on hive.* to hive@'%' identified by 'hive';

• 5.2安装mysql jdbc 驱动

[root@node-1 ~]# yum install mysql-connector-java

6.安装ambari

• 6.1在node-1上安装ambari-server

[root@node-1 ~]# yum install -y ambari-server

• 6.2配置ambari

[root@cluster-master ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? 
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
jdk-8u112-linux-x64.tar.gz... 100% (174.7 MB of 174.7 MB)
Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
Installing JDK to /usr/jdk64/
Successfully installed JDK to /usr/jdk64/
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip to /var/lib/ambari-server/resources/jce_policy-8.zip

Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/jce_policy-8.zip
Installing JCE policy...
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? 
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): 
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Configuring ambari database...
Enter Database Password (bigdata): 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? 

• 6.3 将Ambari数据库脚本导入到数据库

[root@node-1 ~]# mysql
MariaDB [ambari]> use ambari;
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

• 6.4 enter

Proceed with configuring remote database connection properties [y/n] (y)? 
Extracting system views...
ambari-admin-2.6.1.0.143.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@cluster-master ~]# 

• 6.5 启动

 

未完待续。。。

 

 

参考：

http://blog.51cto.com/tryingstuff/2066561

https://www.cnblogs.com/boyzgw/p/6525313.html