ambari+hdp安装集群

最新推荐文章于 2024-06-25 09:00:00 发布

独狐游清湖

最新推荐文章于 2024-06-25 09:00:00 发布

阅读量422

点赞数

分类专栏： hadoop 文章标签：大数据 hadoop

本文链接：https://blog.csdn.net/qq_42422698/article/details/113888653

版权

hadoop 专栏收录该内容

5 篇文章 0 订阅

订阅专栏

一各节点改名（改后重新登录）
hostnamectl set-hostname node104.data
hostnamectl set-hostname node116.data
hostnamectl set-hostname node117.data
hostnamectl set-hostname node118.data
hostnamectl set-hostname node119.data

yum -y install lrzsz 各服务器分别安装rz&sz服务上传下载文件

二各节点编辑hosts
vim /etc/hosts
10.126.8.104 node104.data
10.126.8.116 node116.data
10.126.8.117 node117.data
10.126.8.118 node118.data
10.126.8.119 node119.data

扩大集群
10.126.8.104 node104.data
10.126.8.116 node116.data
10.126.8.117 node117.data
10.126.8.118 node118.data
10.126.8.119 node119.data
10.126.8.173 node173.data
10.126.8.174 node174.data
10.126.8.175 kafka175.data
10.126.8.176 kafka176.data
10.126.8.177 kafka177.data
10.126.8.178 redis178.data
10.126.8.179 redis179.data

三免密
(10.126.8.104、10.126.8.116、10.126.8.117、10.126.8.118和10.126.8.119 )
全部节点执行
rm -rf ~/.ssh

生成密钥对

ssh-keygen -t rsa ## 一路回车即可

进入.ssh目录

cd ~/.ssh

将公钥导入至authorized_keys

cat id_rsa.pub >> authorized_keys

修改文件权限

chmod 700 ~/.ssh
chmod 600 authorized_keys

node104执行:
cat ~/.ssh/id_rsa.pub | ssh root@node116.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node117.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node118.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node119.data ‘cat - >> ~/.ssh/authorized_keys’

node116执行:
cat ~/.ssh/id_rsa.pub | ssh root@node104.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node117.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node118.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node119.data ‘cat - >> ~/.ssh/authorized_keys’

node117执行:
cat ~/.ssh/id_rsa.pub | ssh root@node104.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node116.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node118.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node119.data ‘cat - >> ~/.ssh/authorized_keys’

node118执行:
cat ~/.ssh/id_rsa.pub | ssh root@node104.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node116.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node117.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node119.data ‘cat - >> ~/.ssh/authorized_keys’

node119执行:
cat ~/.ssh/id_rsa.pub | ssh root@node104.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node116.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node117.data ‘cat - >> ~/.ssh/authorized_keys’
cat ~/.ssh/id_rsa.pub | ssh root@node118.data ‘cat - >> ~/.ssh/authorized_keys’

#查看公钥私钥
cat ~/.ssh/authorized_keys
实在不会就手动复制-呼呼
四 jdk
下载链接: https://pan.baidu.com/s/1rlqZejpZZqio9RPzgnGOEg 提取码: j47n ；内有jdk-8u151-linux-x64.tar.gz和mysql-connector-java.jar文件。

mkdir /usr/java；将下载的压缩包上传到java文件夹内
scp -r jdk-8u151-linux-x64.tar.gz root@192.168.112.104:/usr/java/
解压压缩包：cd /usr/java
tar zxvf jdk-8u151-linux-x64.tar.gz
配置jdk环境变量：

编辑vi /etc/profile,文末插入以下内容：

set java

export JAVA_HOME=/usr/java/jdk1.8.0_151
export PATH= $JAVA_HOME/bin:$ PATH
使环境变量生效：source /etc/profile
java -version

五. Mysql部署
1 # 下载mysql5.7的rpm包
wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
2 # 安装第一步下载的rpm文件，安装成功后目录下会增加两个文件
yum -y install mysql57-community-release-el7-11.noarch.rpm

查看mysql57的安装源是否可用，如不可用请自行修改配置文件（/etc/yum.repos.d/mysql-community.repo）使mysql57下面的enable=1

若有mysql其它版本的安装源可用，也请自行修改配置文件使其enable=0

yum repolist enabled | grep mysql
3. 安装mysql
yum install mysql-community-server
4. 设置mysql

启动mysql服务

service mysqld start

查看root密码

grep “password” /var/log/mysqld.log gpxPg=GuT7gx

登陆mysql

mysql -u root -p
Enter password:

为了可以设置简单密码

set global validate_password_policy=0;
set global validate_password_length=7;

立即修改密码，执行其他操作报错：

SET PASSWORD FOR ‘root’@‘localhost’ = PASSWORD(’******’);

我们创建密码为root 123456

新增ambari用户并增加权限
mysql -uroot -p123456
CREATE USER ‘ambari’@’%’ IDENTIFIED BY ‘ambari’;
GRANT ALL PRIVILEGES ON . TO ‘ambari’@’%’;
CREATE USER ‘ambari’@‘localhost’ IDENTIFIED BY ‘ambari’;
GRANT ALL PRIVILEGES ON . TO ‘ambari’@‘localhost’;
CREATE USER ‘ambari’@‘node104.data’ IDENTIFIED BY ‘ambari’;
CREATE USER ‘ambari’@‘10.126.8.104’ IDENTIFIED BY ‘bigdata’;
GRANT ALL PRIVILEGES ON . TO ‘ambari’@‘node104.data’; //本地主机名
GRANT ALL PRIVILEGES ON . TO ‘ambari’@‘10.126.8.104’; //本地主机名
使用ambari用户登陆并创建数据库
mysql -h 10.126.8.104 -uambari -pambari
mysql -uambari -pambari
CREATE DATABASE ambari;
exit;

六时钟同步

七编辑ambari—repo

安装httpd和wget服务(第四步已安装)

安装httpd

yum -y install httpd.x86_64
systemctl enable httpd.service
systemctl start httpd.service

安装wget

yum -y install wget
2. 下载ambari和hdp包

将tar包下载到/var/www/html

cd /var/www/html
wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.1.0/ambari-2.7.1.0-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.1.0/HDP-3.0.1.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

解压上面三个包

tar zxvf ambari-2.7.1.0-centos7.tar.gz
tar zxvf HDP-3.0.1.0-centos7-rpm.tar.gz
tar zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz
3. 新建repo文件
这个很重要。最后在浏览器输入10.126.8.104查看repo位置。
新建ambari.repo文件
[ambari]
name=ambari
baseurl=http://10.126.8.104/ambari/centos7/2.7.1.0-169/
enabled=1
gpgcheck=0
新建HDP.repo文件

[HDP]
name=HDP
baseurl=http://10.126.8.104/HDP/centos7/3.0.1.0-187/
path=/
enabled=1
gpgcheck=0
新建HDP-UTILS.repo文件

[HDP-UTILS]
name=HDP-UTILS
baseurl=http://10.126.8.104/HDP-UTILS/centos7/1.1.0.22/
path=/
enabled=1
gpgcheck=0
将以上文件放入/etc/yum.repos.d/目录下。

scp -r /etc/yum.repos.d/ambari.repo root@10.126.8.116:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/ambari.repo root@10.126.8.117:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/ambari.repo root@10.126.8.118:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/ambari.repo root@10.126.8.119:/etc/yum.repos.d/

scp -r /etc/yum.repos.d/HDP.repo root@10.126.8.116:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/HDP.repo root@10.126.8.117:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/HDP.repo root@10.126.8.118:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/HDP.repo root@10.126.8.119:/etc/yum.repos.d/

scp -r /etc/yum.repos.d/HDP-UTILS.repo root@10.126.8.116:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/HDP-UTILS.repo root@10.126.8.117:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/HDP-UTILS.repo root@10.126.8.118:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/HDP-UTILS.repo root@10.126.8.119:/etc/yum.repos.d/

八主节点安装ambari-server

安装
yum -y install ambari-server
将mysql-connector-java.jar包拷贝到/usr/share/java目录下cp /root/apps/mysql-connector-java.jar java/
修改配置文件
echo server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar >> /etc/ambari-server/conf/ambari.properties
安装ambari-server
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
ambari-server setup 默认下一步Y.需要输入JDK路径echo $JAVA_HOME ：/usr/java/jdk1.8.0_151
初始化数据库
mysql -uambari -pbigdata
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
cd /etc/ambari-server/conf/
vim ambari.properties 可以查看mysql-connector-java.jar路径,证明配置成功。
启动ambari-server
ambari-server start
查看启动日志：
tail -f /var/log/ambari-server/ambari-server.log
如果启动过程失败，可以去 cat /var/log/ambari-server/ambari-server-command.log查看报错信息，一般是由于mysql数据库配置不好导致ambari启动失败。

九注意
1 mysql密码修改
mysql -uroot -p123456
use mysql;show tables;
update mysql.user set authentication_string=‘bigdata’ where User=‘ambari’;
quit;
2 重启mysql
systemctl restart mysqld.service(重启mysql)

十集群安装配置信息
节点信息 8g/4c
10.126.8.104
10.126.8.116
10.126.8.117
10.126.8.118
10.126.8.119

u/p:root/VQK&Jp5z

集群信息：

Admin Name : admin
Cluster Name : EduCluster
Total Hosts : 5 (5 new)
Repositories:
redhat7 (HDP-3.0):
http://10.126.8.104/HDP/centos7/3.0.1.0-187/
redhat7 (HDP-UTILS-1.1.0.22):
http://10.126.8.104/HDP-UTILS/centos7/1.1.0.22/
Services:
HDFS
DataNode : 3 hosts
NameNode : node104.data
NFSGateway : 0 host
SNameNode : node116.data
YARN + MapReduce2
Timeline Service V1.5 : node116.data
NodeManager : 3 hosts
ResourceManager : node104.data
Timeline Service V2.0 Reader : node119.data
Registry DNS : node119.data
Tez
Clients : 3 hosts
Hive
Metastore : node116.data
HiveServer2 : node118.data
Database : New MySQL Database
HBase–已卸载
Master : node117.data
RegionServer : 3 hosts
Phoenix Query Server : 0 host
Sqoop–已卸载
Clients : 3 hosts
ZooKeeper
Server : 5 hosts
Ambari Metrics
Metrics Collector : node117.data
Grafana : node104.data
Kafka
Broker : node119.data
SmartSense
Activity Analyzer : node104.data
Activity Explorer : node104.data
HST Server : node104.data
Spark2
Livy for Spark2 Server : 0 host
History Server : node118.data
Thrift Server : 2 hosts