CDH(Cloudera’s Distribution, including Apache Hadoop),是Hadoop众多分支中的一种,由Cloudera维护,基于稳定版本的Apache Hadoop构建,并集成了很多补丁,可直接用于生产环境。
CDH优点:
1、提供基于web的用户界面,操作方便
2、集成的组件丰富,不需要担心版本问题
3、搭建容易,运维比原生hadoop方便
CDH分为Cloudera Manager管理平台和CDH parcel(parcel包含各种组件的安装包),需要先安装CM,再安装parcel
版本说明:
CM 6.3.1,CDH 6.3.2 (6.3.3版本及以上的要收费了)
组件版本:
https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_63_packaging.html#cdh_630_packaging
相关包的下载地址:
查看自己的系统版本,命令: cat /proc/version
Cloudera Manager 6.3.1下载地址:
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
CDH 6.3.2下载地址:
https://archive.cloudera.com/cdh6/6.3.2/parcels/,由于我们的操作系统为CentOS 7,需要下载以下红框内的文件:
环境说明
IP | HostName | 安装用户 | 备注 |
---|---|---|---|
主节点59 | cdh6-master | mysql数据库和CM安装在同一台机器 | |
从节点60 | cdh6-slave1 | ||
从节点61 | cdh6-slave2 | ||
从节点58 | cdh6-slave3 |
hosts配置
修改hosts文件,末尾加配置:
命令:sudo vim /etc/hosts
主节点59的IP cdh6-master
从节点60的IP cdh6-slave1
从节点61的IP cdh6-slave2
从节点58的IP cdh6-slave3
SSH免密登录
防火墙关闭
service iptables stop
chkconfig iptables off
SELINUX关闭
setenforce 0
vi /etc/selinux/config
SELINUX=disabled
安装NTP
yum install ntp 安装
service ntpd start 启动NTP时间服务器
chkconfig ntpd on 设置NTP开机自动启动
chkconfig | grep ntp 查看NTP是否正常运行
JDK安装
CDH对JDK有版本要求,要仔细核对
https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html#concept_ugf_4sg_5gb
cd /usr/java
tar -xvf jdk-8u251-linux-x64.tar.gz
vim ~/.bash_profile
加入内容:
export JAVA_HOME=/usr/java/jdk1.8.0_251
export PATH=$JAVA_HOME/bin:$PATH
export JRE_HOME=$JAVA_HOME/jre
source ~/.bash_profile
验证是否安装成功:java -version
**各节点安装mysql驱动 **
这里安装了mysql8,所以要找mysql8的驱动
# 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
# 解压
tar -zxf mysql-connector-java-5.1.46.tar.gz
# 创建目录
mkdir /usr/share/java/
# 放到指定目录
(一定要改名为mysql-connector-java.jar)
sudo cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
mysql安装
主节点59 cdh6-master安装
参考 https://www.cnblogs.com/wlwl/p/9686809.html
contos7默认安装了mariadb数据库,这里把他移除:yum remove mariadb-libs.x86_64
yum install mysql-server
下载rpm
wget https://dev.mysql.com/get/mysql80-community-release-el7-1.noarch.rpm
yum localinstall ./mysql80-community-release-el7-1.noarch.rpm
yum install mysql-community-server
安装完成
如有需要,可以手动初始化
mysqld --initialize
启动或关闭mysql服务
service mysqld start --启动mysql
service mysqld stop --关闭mysql
# 查看临时密码
grep 'temporary password' /var/log/mysqld.log
# 登录
mysql -uroot -p
# 输入刚才的密码
ALTER USER 'root'@'localhost' IDENTIFIED BY 'root@1234';
# 新建scm用户
CREATE USER 'scm'@'%' IDENTIFIED BY 'scm@12345';
grant all privileges on *.* to 'scm'@'%' with grant option;
flush privileges;
一些异常处理:
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
# 修改
mysql> set global validate_password.policy=0;
新建CDH所需的数据库:
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
开放权限:
grant all privileges on *.* to 'root'@'%' identified by 'password' with grant option;
grant all privileges on *.* to 'scm'@'master.cdh6' identified by 'password' with grant option;
flush privileges;
master主节点安装cloudera manager server
之前已经下载好了,进行本地安装
sudo yum localinstall ./cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm ./
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-server -y
连接数据库配置
sudo vim /etc/cloudera-scm-server/db.properties
master节点上执行脚本
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
slave从节点安装cloudera manager server
之前已经下载好了,进行本地安装
sudo yum localinstall ./cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm ./cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
cloudera-manager-agent -y
参考资料:cloudera官网教程https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/installation.html
友情链接:
https://blog.csdn.net/qq_15973399/article/details/100043380
https://www.cnblogs.com/dengbangpang/archive/2020/07/02/13224244.html
https://blog.csdn.net/weixin_41624160/article/details/103493113