CentOS7.5,CDH6安装部署

最新推荐文章于 2024-06-12 18:02:46 发布

bug搬运攻城狮

最新推荐文章于 2024-06-12 18:02:46 发布

阅读量1.2w

点赞数 1

分类专栏： hadoop

本文链接：https://blog.csdn.net/TXBSW/article/details/84648269

版权

hadoop 专栏收录该内容

0 篇文章 0 订阅

订阅专栏

集群规划

1.CDH简介

Cloudera’s Distribution, including Apache Hadoop
基于Web的用户界面,支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop,简化了大数据平台的安装、使用难度。

2.硬件配置

*-u root -p 123456

IP	HostName	OS	Cores	Memory	Disk	Remark
172.20.32.1	master	CentOS7.3	4	32G	3T	NameNode，cloudera Server
172.20.32.2	slave1	CentOS7.3	4	32G	1.5T	NameNode，cloudera agent
172.20.32.3	slave2	CentOS7.3	2	16G	6.6T	DataNode，cloudera agent

注意：假如是虚拟机，磁盘挂载单目录（无需分盘）

其中namenode用来部署HA，其中mysql,hdfs.namenode，Hbase.Master,Yarn.ResourceManager等
datanode用来作为worker节点，还需承担zk，kafka等其他服务的部署

软件配置

JDK1.8.0_131
mysql5.6

172.20.32.37 3306 
-u cdh6 -p 123456

name	版本	来源
CM	6.0.1	https://archive.cloudera.com/cm6/6.0.1/
CDH	6.0.1	https://archive.cloudera.com/cdh6/6.0.1/parcels/
hadoop	3.0.0-cdh6.0.1	cdh自带包
hive	2.1.1-cdh6.0.1	cdh自带包
hbase	2.0.0-cdh6.0.1	cdh自带包
solr	7.0.0+cdh6.0.1	cdh自带包
impala	3.0.0-cdh6.0.1	cdh自带包
hue	3.9.0+cdh6.0.1	cdh自带包
sqoop	1.4.7-cdh6.0.1	cdh自带包
kafka	1.0.0-cdh6.0.1	cdh自带包
spark	2.2.0+cdh6.0.1	cdh自带包
oozie	5.0.0-beta1-cdh6.0.1	cdh自带包
kudu	1.6.0-cdh6.0.1	cdh自带包
flume-ng	flume-ng	cdh自带包

注意：其他组件需要单独安装部署，eg:flink,kylin,ELK等

基础环境配置准备

1.关闭防火墙[master,slave1-2]

临时关闭防火墙
# systemctl stop firewalld
永久防火墙开机自启动
# systemctl disable firewalld 
查看防火墙状态
# systemctl status firewalld

#  systemctl status firewalld
Unit firewalld.service could not be found.

2.selinux关闭[master,slave1-2]

# vim /etc/sysconfig/selinux
SELINUX=disabled
重启机器，命令：
# reboot
重启后检查：
# sestatus –v

3.hostname config[master,slave1-2]

master上执行
# vim /etc/hosts
172.20.32.2 slave1
172.20.32.1 master
172.20.32.3 slave2

# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=slave1
# ---------------------------
slave1上执行
# vim /etc/hosts
172.20.32.2 slave1
172.20.32.1 master
172.20.32.3 slave2

# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=master
# -----------------------------
slave2上执行
# vim /etc/hosts
172.20.32.2 slave1
172.20.32.1 master
172.20.32.3 slave2

# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=slave2
# --------------------------

4.NTP服务器设置[master,slave1-2]

一般默认即可！如有问题还请查阅相关资料

5.安装jdk[master,slave1-2]

安装之前先卸载CentOS7自带的JDK

查看系统已经装的jdk： 
# rpm -qa|grep jdk
卸载jdk：
# rpm -e --nodeps java-1.6.0-openjdk-javadoc-1.6.0.0-1.66.1.13.0.el6.x86_64

1.下载“jdk-8u131-linux-x64.tar.gz”，放到/usr/java/目录下
2.解压
# tar -xzvf jdk-8u131-linux-x64.tar.gz
3.在任何目录下均可编辑  
# vim /etc/profile.d/java_home.sh
export JAVA_HOME=/usr/java/jdk1.8.0_131
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib

# source /etc/profile.d/java_home.sh

注意！必须在该目录下安装JDK，后续cdh默认加载此路径的JDK**

6. 配置各节点之间ssh免密登录[master,slave1-2]

此步骤交给运维去做

1.  CentOS默认没有启动ssh无密登录，
# vim /etc/ssh/sshd_config
去掉其中3行的注释，每台服务器都要置
开启  RSAAuthentication yes
开启  PubkeyAuthentication yes
开启  AuthorizedKeyFile  .ssh/authorized_keys
  
# ssh-keygen -t rsa
一直按回车
# ls /root/.ssh/ 
id_rsa 和 id_rsa.pub
# cp id_rsa.pub authorized_keys
 两个woker节点执行        
# cat ~/.ssh/authorized_keys | ssh root@172.20.32.1 'cat >> ~/.ssh/authorized_keys'

master上执行
# scp authorized_keys root@172.20.32.2:/root/.ssh/
# scp authorized_keys root@172.20.32.3:/root/.ssh/
# scp known_hosts root@172.20.32.2:/root/.ssh/
# scp known_hosts root@172.20.32.3:/root/.ssh/
slave执行
# chmod 700 ~/.ssh
# chmod 600 ~/.ssh/authorized_keys
重启ssh服务
# systemctl restart sshd.service
  验证能否无密码ssh，在master服务器上执行操作 
# ssh slave1
不同用户之间退出
# logout

7.优化虚拟内存需求率[master,slave1-2]

检查虚拟内存需求率
# cat /proc/sys/vm/swappiness
显示如下：
 30
临时降低虚拟内存需求率
# sysctl vm.swappiness=10
永久降低虚拟内存需求率
# echo 'vm.swappiness = 10' > /etc/sysctl.d/swappiness.conf
并运行如下命令使生效
# sysctl -p

8. 解决透明大页面问题[master,slave1-2]

检查透明大页面问题
# cat /sys/kernel/mm/transparent_hugepage/defrag
如果显示为：
[always] madvise never
临时关闭透明大页面问题
# echo never > /sys/kernel/mm/transparent_hugepage/defrag

确认配置生效
# cat /sys/kernel/mm/transparent_hugepage/defrag
应该显示为：
always madvise [never]
配置开机自动生效
# echo never > /sys/kernel/mm/transparent_hugepage/enabled

9.mysql安装

FAQ
采用的是DBA提供的172.20.32.37 这个服务器上的mysql

10 拷贝mysql-connector-java到各个节点指定目录下[master,slave1-2]

[root@master ~]# cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar

11.创建数据库[172.20.32.37]

> mysql -u root -p 123456
CREATE DATABASE db_cdh6_amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE db_cdh6_hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE db_cdh6_metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE db_cdh6_sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE db_cdh6_oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci
CREATE DATABASE db_cdh6_hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;


GRANT ALL ON db_cdh6_scm.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON db_cdh6_amon.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON db_cdh6_hue.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON db_cdh6_metastore.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON db_cdh6_sentry.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON db_cdh6_oozie.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON db_cdh6_hive.* TO 'cdh6'@'%' IDENTIFIED BY '123456';

FLUSH PRIVILEGES;

Cloudera Manager安装

1、配置 Cloudera Manager 仓库master,slave1-2

[root@master ~]# wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ 
[root@master ~]# rpm --import https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPM-GPG-KEY-cloudera

[root@slave1 ~]# wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ 
[root@slave1 ~]# rpm --import https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPM-GPG-KEY-cloudera

[root@slave2 ~]# wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ 
[root@slave2 ~]# rpm --import https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPM-GPG-KEY-cloudera

使用仓库安装会比较慢，建议先把需要的rpm下载下来，进行离线安装或者建私有仓库，主要下面的三个软件包：

cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm

//cloudera-manager-daemons 是 server 和 agent 必须安装的。

构建本地仓库(2) [任意服务器]

master

cm6.0.1
[root@slave3 ~]# sudo yum install httpd

[root@slave3 ~]# sudo systemctl start httpd

[root@slave3 ~]# sudo mkdir -p /var/www/html/cloudera-repos

[root@slave3 ~]# sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cm6/6.0.1/redhat7/ -P /var/www/html/cloudera-repos

[root@slave3 ~]# sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm6

- cdh6.0.1

[root@slave3 ~]# sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.0.1/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel -P /var/www/html/cloudera-repos

[root@slave3 ~]# sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.0.1/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256 -P /var/www/html/cloudera-repos

[root@slave3 ~]# sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/cdh6/6.0.1/parcels/manifest.json -P /var/www/html/cloudera-repos

[root@slave3 ~]# sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/6.0.1/redhat7/ -P /var/www/html/cloudera-repos

[root@slave3 ~]# sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6

创建访问端口

[root@slave3 ~]# cd /var/www/html

[root@slave3 ~]# python -m SimpleHTTPServer 8900

构建yum源[master,slave1-2]

[root@master ~]# vim /etc/yum.repos.d/cloudera-repo.repo
[root@slave1 ~]# vim /etc/yum.repos.d/cloudera-repo.repo
[root@slave2 ~]# vim /etc/yum.repos.d/cloudera-repo.repo

[cloudera-repo]
name=cloudera-repo
baseurl=http://master:8900/cloudera-repos/cm6/6.0.1/redhat7/yum/
enabled=1
gpgcheck=0

2.安装 CM Server 和 Agent

建议离线安装，把rpm包下载到服务器上面，传到其他节点一份，再本地安装，速度会快很多。

master：

[root@master ~]# yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

slave1,slave2：

[root@slave1 ~]# yum install cloudera-manager-daemons cloudera-manager-agent
[root@slave2 ~]# yum install cloudera-manager-daemons cloudera-manager-agent

3.设置 Cloudera Manager 数据库

mysql在本地时执行：
[root@master ~]# rm -rf //cloudera-scm-server/db.mgmt.properties
[root@master ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql db_cdh6_scm -h172.20.32.37  -uroot -p'blf1#root' --scm-host 172.20.32.1 db_cdh6_scm scm scm

JAVA_HOME=/usr/java/jdk1.8.0_131
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

注意！上面最好指定ip,切勿使用hostname很容易出现问题

4.安装 CDH[master]

配置CDH的软件包 parcels(namenode01)
CM安装成功之后，接下来我们就可以通过CM安装CDH的方式构建企业大数据平台。所以首先需要把CDH的parcels包下载到CM主服务器上。
同样的，我们为了加速我们的安装，我们可以把需要下载的软件包提前下载下来，也可以创建CDH私有仓库。

[root@master ~]# cd /opt/cloudera/parcel-repo
[root@master ~]# wget http://172.20.32.36/package/CDH/CDH-6.0.1-1/CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
[root@master ~]# wget https://archive.cloudera.com/cdh6/6.0.1/parcels/manifest.json
[root@master ~]# wget https://archive.cloudera.com/cdh6/6.0.1/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256

[root@master ~]# mv CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha256 CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha

也可以执行下面的操作
在manifest.json文件中，找到对应版本的秘钥，复制到.sha文件中。

[root@master ~]# echo "2e650f1f1ea020a3efc98a231b85c2df1a50b030" > CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha

修改属主属组。

[root@master ~]# chown cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/ *

启动 Cloudera Manager Server[master]

[root@master ~]# systemctl start cloudera-scm-server

如果启动中有什么问题，可以查看日志。[master]

[root@master ~]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

参考：
http://blog.51cto.com/wzlinux/2321433
https://www.cloudera.com/documentation/enterprise/6/6.0/topics/installation.html

bug搬运攻城狮

关注

1
点赞
踩
26

收藏

觉得还不错? 一键收藏
0
评论
CentOS7.5,CDH6安装部署

SCM-CDH安装部署集群规划1.CDH简介Cloudera’s Distribution, including Apache Hadoop基于Web的用户界面,支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop,简化了大数据平台的安装、使用难度。2.硬件配置-u root -p 123456IP...
复制链接

扫一扫

专栏目录