hdp和mysql集群_Ambari2.7.3 和HDP3.1.0搭建Hadoop集群

一、环境及软件准备

1、集群规划

hdp01/10.1.1.11hdp02/10.1.1.12hdp03/10.1.1.13hdp04/10.1.1.14hdp05/10.1.1.15

ambari server

是

否

否

否

否

ambari agent

是

是

是

是

是

namenode

是

是

否

否

否

datanode

否

否

是

是

是

resourcemanager

是

是

否

否

否

journalnode

是

是

是

是

是

zookeeper

是

是

是

否

否

01和02主要做管理机，03-05做数据数据节点，01-03做zookeeper集群。

主机环境为Ubuntu server18.04，主机间已经建立互信。

2、下载ambari安装包

image.png

ps需要下载repo file,ambari.list

3、下载hdp安装包

image.png

ps需要下载repo file，hdp.gpl.list，hdp.list.

4、安装一台lnmp备用

选择hadoop01

wget -c http://soft.vpser.net/lnmp/lnmp1.5.tar.gz && tar zxf lnmp1.5.tar.gz && cd lnmp1.5 && ./install.sh lnmp

选择安装mysql5.7版本用于支持ambari

开启远程访问mysql。

mysql -u root -p

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' ;

flush privileges;

5、复制下载文件

放到lnmp的默认目录下。

mkdir /home/wwwroot/default/ambari

mkdir /home/wwwroot/default/ambari/HDP-UTILS

cd /home/wwwroot/default/ambari

复制下载文件到相关目录并解压。

tar -zxvf ambari-2.7.3.0-ubuntu18.tar.gz

tar -zxvf HDP-GPL-3.1.0.0-ubuntu18-gpl.tar.gz

tar -zxvf HDP-3.1.0.0-ubuntu18-deb.tar.gz

tar -zxvf HDP-UTILS-1.1.0.22-ubuntu18.tar.gz -C HDP-UTILS

开启nginx目录浏览

nginx.conf http { }加入

autoindex on; #开启nginx目录浏览功能

autoindex_exact_size off; #文件大小从KB开始显示

autoindex_localtime on;

image.png

6、配置本地源

cd /etc/apt/sources.list.d

ambari.list

#VERSION_NUMBER=2.7.3.0-139

#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json

deb http://192.168.0.151/ambari/ambari/ubuntu18/2.7.3.0-139/ Ambari main

hdp.list

#VERSION_NUMBER=3.1.0.0-78

deb http://192.168.0.151/ambari/HDP/ubuntu18/3.1.0.0-78/ HDP main

deb http://192.168.0.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu18/1.1.0.22/ HDP-UTILS main

hdp.gpl.list

#VERSION_NUMBER=3.1.0.0-78

deb http://192.168.0.151/ambari/HDP-GPL/ubuntu18/3.1.0.0-78/ HDP-GPL main

更新list

sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD

sudo apt-get update

apt-cache showpkg ambari-server

apt-cache showpkg ambari-agent

apt-cache showpkg ambari-metrics-assembly

复制到各台主机,并在各台主机更新list。

scp /etc/apt/sources.list.d/ambari.list 192.168.0.152:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/ambari.list 192.168.0.153:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/ambari.list 192.168.0.154:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/ambari.list 192.168.0.155:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.list 192.168.0.152:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.list 192.168.0.153:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.list 192.168.0.154:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.list 192.168.0.155:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.152:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.153:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.154:/etc/apt/sources.list.d/

scp /etc/apt/sources.list.d/hdp.gpl.list 192.168.0.155:/etc/apt/sources.list.d/

二、安装hdp集群

1、安装ambari &hadoop01

sudo apt-get install ambari-server

Success. You can now start the database server using:

/usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start

Ver Cluster Port Status Owner Data directory Log file

10 main 5432 down postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log

update-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode

Setting up postgresql (10+190) ...

Setting up ambari-server (2.7.3.0-139) ...

Processing triggers for systemd (237-3ubuntu10.9) ...

Processing triggers for ureadahead (0.100.0-20) ...

Processing triggers for libc-bin (2.27-3ubuntu1) ...

2、配置ambari

root@hadoop01:/home/wwwroot/default/ambari# sudo ambari-server setup

Using python /usr/bin/python

Setup ambari-server

Checking SELinux...

WARNING: Could not run /usr/sbin/sestatus: OK

Customize user account for ambari-server daemon [y/n] (n)? y

Enter user account for ambari-server daemon (root):

Adjusting ambari-server permissions and ownership...

Checking firewall status...

Checking JDK...

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Custom JDK

==============================================================================

Enter choice (1):

配置java路径即可

/usr/lib/jvm/java-8-openjdk-amd64

配置mysql

Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.

Checking GPL software agreement...

GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html

Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)?

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

==============================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL / MariaDB

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

[6] - SQL Anywhere

[7] - BDB

==============================================================================

Enter choice (1): 3

配置jdbc,需要上传jdbc.bin.jar

路径如下。

/var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar

Configuring ambari database...

Enter full path to custom jdbc driver:

Configuring ambari database...

Enter full path to custom jdbc driver: /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar

Copying /var/lib/ambari-server/resources/mysql-connector-java-5.1.47-bin.jar to /usr/share/java

Configuring remote database connection properties...

WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Proceed with configuring remote database connection properties [y/n] (y)?

Extracting system views...

....ambari-admin-2.7.3.0.139.jar

Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...

Adjusting ambari-server permissions and ownership...

Ambari Server 'setup' completed successfully.

三、配置ambari

1、导入mysql库

mysql -u root -p

mysql> create database ambari;

mysql> use ambari;

mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

2、启动ambari

root@hadoop01:/home/wwwroot/default/ambari#

sudo ambari-server start

Using python /usr/bin/python

Starting ambari-server

Ambari Server running with administrator privileges.

Organizing resource files at /var/lib/ambari-server/resources...

Ambari database consistency check started...

Server PID at: /var/run/ambari-server/ambari-server.pid

Server out at: /var/log/ambari-server/ambari-server.out

Server log at: /var/log/ambari-server/ambari-server.log

Waiting for server start..................................................

Server started listening on 8080

DB configs consistency check: no errors and warnings were found.

Ambari Server 'start' completed successfully.

四、安装hadoop集群

http://192.168.0.151:8080

用户名密码、admin/admin

image.png

image.png

image.png

选择本地库。

输入目录。

http://192.168.0.151/ambari/HDP/ubuntu18/3.1.0.0-78

http://192.168.0.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu18/1.1.0.22/

image.png

cat ~/.ssh/id_rsa

获取ssh密钥。

image.png

image.png

其他主机出现错误，跟agent没安装上有关。

在其他主机执行

apt install ambari-agent -y

image.png

Ranger、Ranger KMS选了安装失败。取消可以成功。

image.png

image.png