HDP集群搭建

2019独角兽企业重金招聘Python工程师标准>>>

下载包:

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.0.0/ambari-2.6.0.0-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

//如果出现以下结果说明启动了THP

[root@node-41 local]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never

[root@node-41 local]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

//永久关闭THP

[root@node-41 local]# vi /etc/rc.d/rc.local 

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
        echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

修改文件测执行权限：

[root@node-41 ~]# chmod +x /etc/rc.d/rc.local

//检查是否生效

reboot

重启后查看

[root@node-41 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@node-41 ~]# cat /sys/kernel/mm/transparent_hugepage/defrag 
always madvise [never]

说明关闭成功

二、克隆机器及其配置

在VMware中克隆5台及其：node-42、node-43、node-44、node-45、node-46

点击每台机器中的“网络适配器”->“高级”->"生成"（mac地址）->“确定”->"确定"

开机，修改IP地址： vi /etc/sysconfig/network-scripts/ifcfg-ens33 

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=62acf16a-30fd-4909-a68b-92c63408b5b2
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.194.46
GATEWAY=192.168.194.2
NETMASK=255.255.255.0
DNS1=8.8.8.8

修改hostname：

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# 
[root@localhost ~]# hostnamectl set-hostname node-46
[root@localhost ~]# hostname
node-46

关闭防火墙：

//临时关闭

[root@node-42 ~]# systemctl stop firewalld

//禁止开机启动
[root@node-42 ~]# systemctl disable firewalld

//查看防火墙状态
[root@node-42 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

安装Mysql，具体可见 https://my.oschina.net/u/3491962/blog/1934088

修改mysql配置文件：[root@node-41 local]# vi /etc/my.cnf 

[mysqld] 下添加：

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
collation_server=utf8_general_ci
character_set_server=utf8
default-storage-engine=INNODB

[client]
default-character-set=utf8

重启MySQL服务：
[root@node-41 local]# systemctl restart mysqld

连接Mysql查看编码是否替换：
mysql> show variables like '%character_set_%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

创建ambri相应的DB以及其用户名：
mysql> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'ambari'@'%' IDENTIFIED BY 'Ambari123'; 
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

创建HiveDB 以及用户：

mysql> create database hive character set utf8;        
Query OK, 1 row affected (0.00 sec)

mysql> create user 'hive'@'%' IDENTIFIED BY 'Hive123';             
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';            
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

安装Mysql-java

[root@node-41 local]# yum install mysql-connector-java

配置免密登陆：

在node-41 中：

[root@node-41 local]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Ri8XZHtAAQUTWADi5zcW2qRUsDEqJKto/V0MXJH2ULk root@node-41
The key's randomart image is:
+---[RSA 2048]----+
|... =oo+*O@o.    |
|oo o =o .B +     |
|o o + ooo = o    |
|o..+ = oo. E     |
|o. .+ = Soo      |
|.   .o.o.o       |
|     . .         |
|                 |
|                 |
+----[SHA256]-----+

将公钥拷贝至node-41、node-42、node-43、node-44、node-45

ssh-copy-id node-41、ssh-copy-id node-42、ssh-copy-id node-43、ssh-copy-id node-44、ssh-copy-id node-45

在node-42 中：执行同样的操作（生成公钥、私钥，然后将公钥拷贝至各个机器上）

二、配置本地的yum源

在本地yum源上安装相关yum工具

[root@yum ~]# yum install yum-utils -y

[root@yum ~]# yum repolist

[root@yum ~]# yum install createrepo -y

安装Apache httpd

[root@yum ~]# yum install httpd -y

安装完成后会自动生成目录：[root@yum ~]# /var/www/html/，相当于Tomcat 的 /webapps目录，进入该目录，创建ambari和hdp文件夹 

[root@yum html]# cd /var/www/html/

[root@yum html]# mkdir ambari
[root@yum html]# mkdir hdp
[root@yum html]# mkdir hdp/HDP-UTILS-1.1.0.21

解压相应的文件：

[root@yum html]# tar -zxvf /root/ambari-2.6.0.0-centos7.tar.gz -C /var/www/html/ambari/

[root@yum html]# tar -zxvf /root/HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.21/

[root@yum html]# tar -zxvf /root/HDP-2.6.5.0-centos7-rpm.tar.gz -C /var/www/html/hdp/

解压完成后启动 httpd 服务：

[root@yum html]# systemctl start httpd

启动后可以访问链接：http://192.168.194.40/

设置开机自启动：

[root@yum html]# systemctl enable httpd 

配置ambari yum 源：

[root@yum html]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari.repo -O /etc/yum.repos.d/ambari.repo

[root@yum yum.repos.d]# vi ambari.repo 

#VERSION_NUMBER=2.6.0.0-267
[ambari-2.6.0.0]
name=ambari Version - ambari-2.6.0.0
baseurl=http://192.168.194.40/ambari/ambari/centos7/2.6.0.0-267/
gpgcheck=1
gpgkey=http://192.168.194.40/ambari/ambari/centos7/2.6.0.0-267/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[root@yum yum.repos.d]# vi HDP.repo 

#VERSION_NUMBER=2.6.5.0-1
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0
baseurl=http://192.168.194.40/hdp/HDP/centos7/
gpgcheck=1
gpgkey=http://192.168.194.40/hdp/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP Version - HDP-UTILS-1.1.0.21
baseurl=http://192.168.194.40/hdp/HDP-UTILS-1.1.0.21/
gpgcheck=1
gpgkey=http://192.168.194.40/hdp/HDP-UTILS-1.1.0.21/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

分发HDP.repo ambari.repo 到各个节点

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-41:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-42:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-43:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-44:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-45:$PWD

生成本地yum源

使用createrepo命令，生成本地源

【createrepo用以创建yum源（软件仓库），即为存放于本机特定位置的众多rpm包建立索引描述各包所需以来信息，并形成源数据】

[root@yum yum.repos.d]# createrepo /var/www/html/hdp/HDP/centos7/

[root@yum yum.repos.d]# createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.21/

 

三、安装 Ambari Server

1、安装服务

[root@node-41 yum.repos.d]# yum install ambari-server

2、设置服务

[root@node-41 yum.repos.d]# ambari-server setup

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk1.8.0_161/
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? Y^H
input not recognized, please try again: 
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): node-41
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Re-enter password: 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.6.0.0.267.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root

[root@node-41 yum.repos.d]# ambari-server setup

四、设置ambari用户远程登陆权限

[root@node-41 yum.repos.d]# mysql -uroot -p123456

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'localhost' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.17 sec)

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.09 sec)

mysql> exit
Bye

使用ambari账号登陆MySQL
[root@node-41 yum.repos.d]# mysql -uambari -pAmbari123 -hnode-41

mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

启动Ambari

[root@node-41 yum.repos.d]# ambari-server start

[root@node-41 yum.repos.d]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

访问Ambari服务：

http://192.168.194.41:8080

安装Agent

在所有节点上安装

yum install ambari-agent

安装成功后可进入Ambari Web端，初始化设置

默认账号密码：

admin

admin

设置集群，点击Launch Install Wizard ：

输入集群名字：

转载于:https://my.oschina.net/u/3491962/blog/1934018