cdh安装-hdfs,yarn,zk,hive,impala

简介

Cloudera hadoop 是众多hadoop 开源发行版中较为活跃与稳定的版本，在hadoop生态各个组件的版本兼容上做了一部分的优化，适合在生产环境中使用，以下简称cdh。

其自带了web管理系统，方便部署角色，监控角色状态，称为Cloudera manager，以下简称cm。

基础组件版本：

 

组件	版本
os	centos7.1
Java	oracle 1.8u181
mysql	5.7.25
cdh	5.16.1
cm	5.16.1

 

角色	host
master/server	172.16.116.170/hdmaster
slave1/agent	172.16.116.171/hdslave1
slave2/agent	172.16.116.172/hdslave2
slave3/agent	172.16.116.173/hdslave3

 

操作系统配置

 

配好/etc/hosts.ssh互信

 

减小交换分区阈值

vi /etc/sysctl.conf  #添加vm.swappiness=10

echo vm.swappiness=10 >> /etc/sysctl.conf

echo 10 >/proc/sys/vm/swappiness

禁用大页

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

关闭防火墙,selinux

service iptables stop

chkconfig iptables off

vi /etc/sysconfig/selinux

  selinux=disabled

 

java 安装

安装在/usr/java/

NTP时钟同步

略

ssh root 用户交换

略

MySQL 安装

略

mysql-connector*.jar 复制到$CM_HOME/share/cmf/lib中

安装cm

官网下载https://archive.cloudera.com/cm5/cm/5/

解压到$CM_HOME=/home/cm

创建cm 管理用户-所有节点

useradd --system --home=/home/cm/cm-5.16.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "cloudera scm user" cloudera-scm

mysql

创建角色存储元数据的数据库与用户

create database scm default character set 'utf8' default collate utf8_general_ci;

...

初始化各个角色数据库-master上执行(mysql 在本机,如果不在,需要用mysql -h mysqlhostname)

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql scm root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql amon root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql rman root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql hue root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql metastore root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql sentry root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql nav root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql navms root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql oozie root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql nms root Gn@123456

/home/cm/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql nas root Gn@123456

 

安装包文件夹创建

server：

mkdir -p /opt/cloudera/parcel-repo

 

所有节点：

mkdir -p /opt/cloudera/parcels

配置节点:

$CM_MANAGER/etc/cloudera-scm-agent/config.ini

# Hostname of the CM server.

server_host=mdw

 

# Port that the CM server is listening on.

server_port=7182

 

分发/etc/hosts,java,cm到所有节点

 

启动cm

 

master:cloudera-scm-server start

所有节点：cloudera-scm-agent start

 

 

web配置

http://mdw:7180   admin/admin

 

安装cdh

下载 cdh

https://archive.cloudera.com/cdh5/parcels/5.16.1/

下载parcel 和对应的sha1文件

修改sha1去掉 sha   mv *sha1 sha

安装parcel

上面是agent节点parcels目录，下面是本地server（如果重用角色）parcels的目录

 

分发pacel,sha,manifest到所有节点

 

master 到上述本地路径

其他agent到parcel 目录

如果没有发现本地版本,重启server/agent

 

配置角色

1,master 角色和其他 slave分开

2,hive 的metastore/hive server2 单独一台 , 和 impala 的catalog,statestore 部署在一起

其他问题

1,hive metastore/hive server2机器 需要mysql-connector到 /opt/cloudera/parcels/CDH/lib/hive/lib

2,impala 需要安装额外的依赖包

  安装本地yum源,删除其他源repo: 

vi /etc/yum.repos.d/abc-impala.repo
[abc-impala]
name=abc-impala
keepcache=1
baseurl=file:///var/cache/yum/impala/
enabled=1
gpgcheck=0

创建/var/cache/yum/impala,上传rpm包

yum install libxml2-2.9.1-6.el7_2.3.x86_64.rpm

yum install libxml2-python-2.9.1-6.el7_2.3.x86_64.rpm

yum localinstall deltarpm-3.6-3.el7.x86_64.rpm

yum localinstall python-deltarpm-3.6-3.el7.x86_64.rpm

yum localinstall createrepo-0.9.9-28.el7.noarch.rpm

 createrepo ./

yum install gcc

yum install python-devel

yum install cyrus-sasl*