Hadoop集群搭建之CDH6.3.2

最新推荐文章于 2023-01-04 17:41:37 发布

abao_gege

最新推荐文章于 2023-01-04 17:41:37 发布

阅读量416

点赞数

文章标签： hadoop 大数据分布式

本文链接：https://blog.csdn.net/abao_gege/article/details/125869144

版权

Hadoop集群搭建之CDH6.3.2

文章目录

Hadoop集群搭建之CDH6.3.2
前言
集群机器资源介绍
- 修改/etc/hosts文件
关闭防火墙，开机不自启动
禁用selinux
ssh免登录
安装JDK
安装MySQL
CM安装

前言

从后端转到了大数据，学习之路起点，Hadoop集群搭建

集群机器资源介绍

系统	ip	hostname	备注
centos7	192.168.211.130	hadoop0	虚拟机
centos7	192.168.211.131	hadoop1	克隆虚拟机
centos7	192.168.211.132	hadoop2	克隆虚拟机

修改/etc/hosts文件

192.168.211.129 node1
192.168.211.130 node2
192.168.211.131 node3

关闭防火墙，开机不自启动

systemctl stop firewalld
systemctl disable firewalld

禁用selinux

vim /etc/selinux/config
修改以下配置
SELINUX=disabled

ssh免登录

1）生成公钥和私钥：

[root@hadoop0 .ssh]$ ssh-keygen -t rsa

然后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）
2）将公钥拷贝到要免密登录的目标机器上

[root@hadoop0 .ssh]$ ssh-copy-id hadoop0
[root@hadoop0 .ssh]$ ssh-copy-id hadoop1
[root@hadoop0 .ssh]$ ssh-copy-id hadoop2

3）重复1和2的操作，配置hadoop1，hadoop2对hadoop0、hadoop1、hadoop2三台服务器免密登录。

安装JDK

（1）在hadoop0的/opt目录下创建module和software文件夹

mkdir module
mkdir software

（2）上传oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm并安装

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile
java -version

java version "1.8.0_181"

（3）分发,并source

[root@hadoop0 software]# scp -r /usr/java/ hadoop102:/usr/
[root@hadoop0 software]# scp -r /usr/java/ hadoop103:/usr/
[root@hadoop0 software]# scp /etc/profile hadoop102:/etc/
[root@hadoop0 software]# scp /etc/profile hadoop103:/etc/
[root@hadoop1 module]# source /etc/profile
[root@hadoop1 module]# java -version
[root@hadoop2 module]# source /etc/profile
[root@hadoop2 module]# java -version

安装MySQL

注意：一定要用root用户操作如下步骤；先卸载MySQL再安装
1）安装包准备
（1）查看MySQL是否安装

rpm -qa|grep -i mysql

（2）如果安装了MySQL，就先卸载

rpm -e --nodeps mysql-libs-5.1.73-7.el6.x86_64

（3）删除阿里云原有MySql依赖

yum remove mysql-libs

（4）下载MySql依赖并安装

yum install libaio
yum -y install autoconf
wget https://downloads.mysql.com/archives/get/p/23/file/MySQL-shared-compat-5.6.24-1.el6.x86_64.rpm
wget https://downloads.mysql.com/archives/get/p/23/file/MySQL-shared-5.6.24-1.el6.x86_64.rpm
rpm -ivh MySQL-shared-5.6.24-1.el6.x86_64.rpm 
rpm -ivh MySQL-shared-compat-5.6.24-1.el6.x86_64.rpm

（5）上传mysql-libs.zip到hadoop101的/opt/software目录，并解压文件到当前目录

[root@hadoop0 software]# yum install unzip
unzip mysql-libs.zip
ls
mysql-libs.zip
mysql-libs

（6）进入到mysql-libs文件夹下

 [root@hadoop0 mysql-libs]# ll

总用量 76048
-rw-r–r–. 1 root root 18509960 3月 26 2015 MySQL-client-5.6.24-1.el6.x86_64.rpm
-rw-r–r–. 1 root root 3575135 12月 1 2013 mysql-connector-java-5.1.27.tar.gz
-rw-r–r–. 1 root root 55782196 3月 26 2015 MySQL-server-5.6.24-1.el6.x86_64.rpm
2）安装MySQL服务器
（1）安装MySQL服务端

[root@hadoop0 mysql-libs]# rpm -ivh MySQL-server-5.6.24-1.el6.x86_64.rpm

（2）查看产生的随机密码

[root@hadoop0 mysql-libs]# cat /root/.mysql_secret
OEXaQuS8IWkG19Xs

（3）查看MySQL状态

[root@hadoop0 mysql-libs]# service mysql status

（4）启动MySQL

[root@hadoop0 mysql-libs]# service mysql start

3）安装MySQL客户端
（1）安装MySQL客户端

[root@hadoop0 mysql-libs]# rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm

（2）链接MySQL（密码替换成产生的随机密码）

[root@hadoop0 mysql-libs]# mysql -uroot -pOEXaQuS8IWkG19Xs

（3）修改密码

mysql>SET PASSWORD=PASSWORD('000000');

（4）退出MySQL

mysql>exit

4）MySQL中user表中主机配置
配置只要是root用户+密码，在任何主机上都能登录MySQL数据库。
（1）进入MySQL

[root@hadoop0 mysql-libs]# mysql -uroot -p000000

（2）显示数据库
mysql>show databases;
（3）使用MySQL数据库
mysql>use mysql;
（4）展示MySQL数据库中的所有表
mysql>show tables;
（5）展示user表的结构
mysql>desc user;
（6）查询user表
mysql>select User, Host, Password from user;
（7）修改user表，把Host表内容修改为%
mysql>update user set host=‘%’ where host=‘localhost’;
（8）删除root用户的其他host
mysql> delete from user where host!=‘%’;
（9）刷新
mysql>flush privileges;
（10）退出
mysql>quit;

CM安装

（1）将mysql-connector-java-5.1.27-bin.jar拷贝到/usr/share/java路径下，并重命名

[root@hadoop0 mysql-libs]# tar -zxvf mysql-connector-java-5.1.27.tar.gz 
[root@hadoop0 mysql-libs]# cd mysql-connector-java-5.1.27
[root@hadoop0 mysql-connector-java-5.1.27]# mv mysql-connector-java-5.1.27-bin.jar mysql-connector-java.jar
[root@hadoop0 mysql-connector-java-5.1.27]# mkdir /usr/share/java
[root@hadoop0 mysql-connector-java-5.1.27]# cp mysql-connector-java.jar /usr/share/java/
[root@hadoop0 mysql-connector-java-5.1.27]# scp -r /usr/share/java/ hadoop1:/usr/share/
[root@hadoop0 mysql-connector-java-5.1.27]# scp -r /usr/share/java/ hadoop2:/usr/share/

集群规划

节点	hadoop0	hadoop1	hadoop2
服务	cloudera-scm-server cloudera-scm-agent	cloudera-scm-agent	cloudera-scm-agent

（2）创建cloudera-manager目录,存放cdh安装文件

[root@hadoop0 mysql-connector-java-5.1.27]# mkdir /opt/cloudera-manager
[root@hadoop0 mysql-connector-java-5.1.27]# cd /opt/software/
[root@hadoop0 software]# tar -zxvf cm6.3.1-redhat7.tar.gz
[root@hadoop0 software]# cd cm6.3.1/RPMS/x86_64/
[root@hadoop0 x86_64]# mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
[root@hadoop0 x86_64]# mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
[root@hadoop0 x86_64]# mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
[root@hadoop0 x86_64]# cd /opt/cloudera-manager/
[root@hadoop0 cloudera-manager]# ll

total 1185876
-rw-r–r-- 1 2001 2001 10483568 Sep 25 2019 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r–r-- 1 2001 2001 1203832464 Sep 25 2019 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r–r-- 1 2001 2001 11488 Sep 25 2019 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

（3）安装cloudera-manager-daemons,安装完毕后多出/opt/cloudera目录

[root@hadoop0 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop0 cloudera-manager]# cd /opt/cloudera
cloudera/         cloudera-manager/ 
[root@hadoop0 cloudera-manager]# cd ..
[root@hadoop0 opt]# scp -r /opt/cloudera-manager/ hadoop102:/opt/
[root@hadoop0 opt]# scp -r /opt/cloudera-manager/ hadoop103:/opt/
[root@hadoop1 ~]# cd /opt/cloudera-manager/
[root@hadoop1 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop2 ~]# cd /opt/cloudera-manager/
[root@hadoop2 cloudera-manager]# rpm -ivh cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm

（4）安装cloudera-manager-agent

[root@hadoop0 cloudera-manager]# yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt
[root@hadoop0 cloudera-manager]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop1 cloudera-manager]# yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt
[root@hadoop1 cloudera-manager]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm 
[root@hadoop2 cloudera-manager]# yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt
[root@hadoop2 cloudera-manager]# rpm -ivh cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

（5）安装agent的server节点

[root@hadoop0 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop0
[root@hadoop1 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop1
[root@hadoop2 cloudera-manager]# vim /etc/cloudera-scm-agent/config.ini
server_host=hadoop2

（6）安装cloudera-manager-server

[root@hadoop0 cloudera-manager]# rpm -ivh cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

（7）上传CDH包导parcel-repo

[root@hadoop0 parcel-repo]# pwd
/opt/cloudera/parcel-repo
[root@hadoop0 parcel-repo]# ll

total 2033432
-rw-r–r-- 1 root root 2082186246 May 21 11:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r–r-- 1 root root 40 May 21 10:56 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
-rw-r–r-- 1 root root 33887 May 21 10:56 manifest.json

[root@hadoop0 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

（8）修改server的db.properties

[root@hadoop0 parcel-repo]# vim /etc/cloudera-scm-server/db.properties 
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop101:3306
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=scm
com.cloudera.cmf.db.setupType=EXTERNAL

（9）启动server服务

[root@hadoop0 log]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
[root@hadoop0 software]# systemctl start cloudera-scm-server

（10）启动agent节点

[root@hadoop0 software]# systemctl start cloudera-scm-agent
[root@hadoop1 software]# systemctl start cloudera-scm-agent
[root@hadoop2 software]# systemctl start cloudera-scm-agent