软件环境:cm 5.6,cdh 5.6
cm 离线安装包下载
http://archive-primary.cloudera.com/cm5/cm/5/
cdh 离线安装包下载
http://archive-primary.cloudera.com/cdh5/parcels/
kakfa 离线安装包下载
http://archive.cloudera.com/kafka/parcels/latest/
cdh离线包只需要放在cm的主节点
1前期准备工作
1.1修改主机名
/etc/sysconfig/network
注意事项:主机名不能带下划线_,可以带中划线-
1.2 ip与hostname绑定
执行命令 vi /etc/hosts
192.168.80.101 master1
192.168.80.201 datanode1
192.168.80.202 datanode2
192.168.80.203 datanode3
1.3设置ssh免密码登陆
//配置主节点到从节点的免密登陆
执行命令 (1)ssh-keygen -t rsa
(2)cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
验证: ssh datanode1
ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave_node_ip
1.4关闭防火墙
$ service iptables off
$ chkconfig iptables off
1.5 关闭 selinux
setenforce 0 (临时生效)
修改 /etc/selinux/config 下的 SELINUX=disabled (重启后永久生效)
1.6 java 环境变量
在文件/etc/profile/中添加以下内容
export JAVA_HOME=/usr/java/jdk1.7
export PATH=
JAVAHOME/bin:
PATH
export CLASSPATH=.:
JAVAHOME/lib/dt.jar:
JAVA_HOME/lib/tools.jar
1.7 配置java全局变量
$ echo "JAVA_HOME=/usr/java/jdk1.7" >> /etc/environment
$ chmod -R /usr/java/jdk1.7
1.8 主从节点日期同步
选择局域网中的一台机器作为ntp服务器
先让ntp 服务器与网络授时组织的中国授时源校时
ntpdate cn.pool.ntp.org
1.8.1 安装 NTP
yum install ntp
1.8.2 配置 ntp 服务器
在ntp服务器的 /etc/ntp.conf 中添加
只能使用内网的情况下
server 127.127.1.0
fudge 127.127.1.0 stratum 10
如果该服务器能够连接外网,可配置为
server cn.pool.ntp.org
fudge cn.pool.ntp.org stratum 10
1.8.3其他节点的时间与ntp服务器同步
在从节点 /etc/ntp.conf 中添加
server
fudge stratum 10
1.8.4 所有节点设置ntpd 随机启动
chkconfig ntpd on
1.8.5 启动ntpd 服务
service ntpd start
1.8.6 synchronize the node
ntpdate -u
1.8.7 Synchronize the system clock (to prevent synchronization problems).
hwclock –systohc
1.9 关闭ipv6
2 cloudera manager 安装
2.1 解压cm
cloudera manager的目录默认位置为/opt
$ tar -zxvf cloudera-manager-el6-cm5.6.0_x86_64.tar.gz -C /opt/
2.2在所有节点创建cloudera-scm用户
sudo useradd –system –home=/opt/cloudera-manager/cm-5.6.0/run/cloudera-scm-server –no-create-home –shell=/bin/false –comment “Cloudera SCM User” cloudera-scm
2.3配置mysql
选用Mysql用于存储cm的配置文件
2.3.1 安装mysql
$ rpm MySQL-server-5.5.48-1.el6.x86_64.rpm
$ rpm MySQL-client-5.5.48-1.el6.x86_64.rpm
安装后启动 mysql服务
$ sudo service mysqld start
2.3.2 拷贝mysql驱动
mysql jdbc (mysql-connector-java.jar)拷贝到下面文件夹
$ cp mysql-connector-java-5.1.38-bin.jar /opt/cm-5.6.0/share/cmf/lib/mysql-connector-java.jar
$ cp mysql-connector-java-5.1.38-bin.jar /usr/share/java/mysql-connector-java.jar
2.3.3 初始化mysql (要先启动mysql)
$ sudo /usr/bin/mysql_secure_installation
[…]
Enter current password for root (enter for none):
OK, successfully used password, moving on…
[…]
Set root password? [Y/n] y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[…]
Disallow root login remotely? [Y/n] N
[…]
Remove test database and access to it [Y/n] Y
[…]
Reload privilege tables now? [Y/n] Y
All done!
2.3.4 把mysql加入开
$ sudo /sbin/chkconfig mysql on
$ sudo /sbin/chkconfig --list mysqlqld
0:off 1:off 2:on 3:on 4:on 5:on 6:off
2-5为on则加入自启成功
2.4 初始化话cm 所用数据库
2.4.1 创建cm的配置存储数据库
//mysql 客户端下,root用户下执行
create database scm DEFAULT CHARACTER SET utf8;
grant all on scm.* TO 'hadoop'@'%' IDENTIFIED BY 'hadoop';
flush privileges;
2.4.2 初始化 cm数据库
在主节点初始化CM的数据库:
$ /opt/cm-5.6.0/share/cmf/schema/scm_prepare_database.sh mysql scm hadoop hadoop
//scm 是数据库名称,hadoop 是数据库账号,123是数据库密码
2.5 cm Agent
2.5.1 修改主机名修改/opt/cm-5.6.0/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名
2.6 分发文件到子节点
2.6.1 拷贝 cm 到子节点
scp -r /opt/ root@datanode1:/opt/
2.7 准备cdh5.6的离线安装包Parcels
将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中(parcel-repo需要手动创建)。
2.8 启动cm
在主节点启动服务端
sudo /opt/cm-5.6.0/etc/init.d/cloudera-scm-server start
在各个节点启动客户端服务
sudo /opt/cm-5.6.0/etc/init.d/cloudera-scm-agent start
2.9 cm 日志
服务端日志
tail -300f /opt/cm-5.6.0/log/cloudera-scm-server/cloudera-scm-server.log
客户端日志
tail -300f /opt/cm-5.6.0/log/cloudera-scm-agent/cloudera-scm-agent.log
3.0 web访问
4 集群安装常见问题
4.1 swappiness 设置为0
sysctl vm.swappiness=0 (临时生效)
echo vm.swappiness=0 >> /etc/sysctl.conf (永久生效)
4.2 关闭透明大页面
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag (临时生效)
echo ‘echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag’ > /etc/rc.local
5 mysql数据库脚本
5.1 hive
mysql> create database hive DEFAULT CHARACTER SET utf8;
mysql> grant all on hive.* TO ‘hadoop’@’%’ IDENTIFIED BY ‘hadoop’;
5.2 Activity Monitor
mysql> create database amon DEFAULT CHARACTER SET utf8;
mysql> grant all on amon.* TO ‘hadoop’@’%’ IDENTIFIED BY ‘hadoop’;
5.3 Oozie Server
mysql>create database oozie DEFAULT CHARACTER SET utf8;
mysql> grant all on oozie.* TO ‘hadoop’@’%’ IDENTIFIED BY ‘hadoop’;
5.4 hue
mysql>create database hue DEFAULT CHARACTER SET utf8;
mysql> grant all on hue.* TO ‘hadoop’@’%’ IDENTIFIED BY ‘hadoop’;
特别说明:hue 服务端是用python开发的,需要安装python的mysql驱动