1.CDH安装前配置
1.1 HOSTS配置
1.1.1 hostname修改
修改配置文件/etc/sysconfig/network
root@hadoop01~# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop01
修改完成后需要重启服务器使其生效。修改集群所有节点的hostname为统一格式方便集群管理。
1.1.2 配置/etc/hosts文件
配置/etc/hosts文件
[root@hadoop01 ~]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.23.216.73 hadoop01
172.23.216.74 hadoop02
172.23.216.75 hadoop03
1.2 免密码登陆配置
SSH免密码登陆，因为HADOOP环境中需要机器传输文件啥的。
1.2.1、在五台机器上均执行命令ssh-keygen -t rsa，遇到提示一路回车，生成公钥文件
1.2.2、在master上面执行命令cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
1.2.3、在master上面将授权文件分别拷贝到其它四个节点scp ~/.ssh/authorized_keys root@hadoop01:~/.ssh/
此时的你可以在master上任意scp一个文件到其它节点机器，发现不用输入密码了，恭喜成功了。
注意这里也有一种比较简单的方式配置免密，在各个节点上生成秘钥之后，可以用ssh-copy-id -i hadoop01,将秘钥拷贝到各个节点
1.3 禁用SELinux
1.3.1 在集群所有节点执行如下命令
[root@hadoop01 ~]# setenforce 0
1.3.2 修改集群所有节点配置文件/etc/selinux/config
[root@hadoop01 ~]# vim /etc/selinux/config
1.3.3 验证是否禁用成功
[root@hadoop01 ~]# sestatus -v
SELinux status: disabled
1.4 关闭防火墙
1.4.1 关闭集群所有服务器的防火墙
[root@hadoop01 ~]# chkconfig iptables off
service ipta[root@hadoop01 ~]#bles stop
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Flushing firewall rules: [ OK ]
iptables: Unloading modules: [ OK ]
1.4.2 检查所有服务器防火墙是否关闭
[root@hadoop01 ~]# service iptables status
iptables: Firewall is not running.
1.5 设置SWAP
1.5.1 在集群所有节点修改/etc/sysctl.conf配置文件，增加如下配置
[root@hadoop01 ~]# vim /etc/sysctl.conf
1.5.2 检查设置是否生效
[root@hadoop01 ~]# sysctl -p
kernel.msgmnb= 65536
kernel.msgmax= 65536
kernel.shmmax= 68719476736
kernel.shmall= 4294967296
vm.swappiness= 10
1.6 关闭透明大页面
1.6.1 在集群所有节点执行如下命令，使其立即生效
[root@hadoop01 ~]#echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
[root@hadoop01 ~]# echo never >/sys/kernel/mm/redhat_transparent_hugepage/enabled
1.6.2 在集群所有节点/etc/rc.d/rc.local脚本中增加如下代码，使其永久生效
if test -f/sys/kernel/mm/redhat_transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
fi
1.6.3 赋予rc.local脚本可执行权限
[root@hadoop01_rc.d# chmod +x /etc/rc.d/rc.local
1.7 配置时钟同步
1.7.1 在集群所有节点安装NTP服务
[root@hadoop01 ~] yum -y install ntp
1.7.2 选择集群一台服务器作为NTP Server，配置如下
[root@hadoop01 ~] vim /etc/ntp.conf
server 127.0.0.1
fudge 127.0.0.1 stratum 10
1.7.3 集群其它节点与其同步，配置如下：
[root@hadoop02 ~]# vim /etc/ntp.conf
server 172.23.216.73
1.7.4 完成以上配置后，在集群所有节点执行如下命令
[root@hadoop01 ~]# chkconfig --add ntpd
[root@hadoop01 ~]# chkconfig ntpd on
[root@hadoop01 ~]# service ntpd restart
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
1.7.5 .验证所有节点NTP是否同步
[root@hadoop01 ~]#ntpq -p
remote refid st t when pollreach delay offset jitter
*ip-172-31-6-148210.23.25.77 2 u 448 1024 377 0.255 3.245 7.859
注：上图标注看到有*号则表示同步成功。
1.8 安装MySQL数据库
1.8.1 在集群中CM节点安装MySQL服务
[root@hadoop01 /]# yum -y install mysql mysql-serve
1.8.2 启动MySQL服务并将mysqld加入开机自启动服务
[root@hadoop01 /]# chkconfig --add mysqld
[root@hadoop01 /]# chkconfig mysqld on
[root@hadoop01 /]# service mysqld start
Starting mysqld: [ OK ]
1.8.3 执行如下脚本初始化MySQL
[root@hadoop01 ~]# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FORALL MySQL
SERVERS IN PRODUCTION USE! PLEASE READEACH STEP CAREFULLY!
In order to log into MySQL to secure it, we'll needthe current
password for the root user. If you'vejust installed MySQL, and
you haven't set the root password yet, the passwordwill be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.
Set root password? Y/n y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them. This is intended only for testing,and to make the installation
go a bit smoother. You should removethem before moving into a
production environment.
Remove anonymous users? Y/n y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? Y/n n
... skipping.
By default, MySQL comes with a database named 'test' that anyone can
access. This is also intended only fortesting, and should be removed
before moving into a production environment.
Remove test database and access to it? Y/n y
- Dropping test database...
... Success!
- Removing privileges on testdatabase...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? Y/n y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MySQL
installationshould now be secure.
Thanks for usingMySQL!
1.8.4 登录MySQL验证是否安装成功
[root@hadoop01 ~]# mysql -uroot -p123456
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013,Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporationand/or its
affiliates.Other names may be trademarks of theirrespective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current inputstatement.
mysql>
1.8.5 创建CDH所需要的数据库
====================hive=============
create database metastore default character set utf8;
CREATE USER 'hive'@'%'IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';
FLUSH PRIVILEGES;
==================hive==========
create database hive default character set utf8;
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
FLUSH PRIVILEGES;
==================cm==========
create database cm default character set utf8;
CREATE USER 'cm'@'%'IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON cm.* TO 'cm'@'%';
FLUSH PRIVILEGES;
============================xxx============
create database am default character set utf8;
CREATE USER 'am'@'%'IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON am.* TO 'am'@'%';
FLUSH PRIVILEGES;
=======================xxx=====
create database rm default character set utf8;
CREATE USER 'rm'@'%'IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON rm.* TO 'rm'@'%';
FLUSH PRIVILEGES;
=======hue====
create database hue default character set utf8;
CREATE USER 'hue'@'%'IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%';
FLUSH PRIVILEGES;
======oozie=====
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
FLUSH PRIVILEGES;
========sentry====
create database sentry default character set utf8;
CREATE USER 'sentry'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON sentry.* TO 'sentry'@'%';
FLUSH PRIVILEGES;
1.9 安装jdk
1.9.1 卸载OpenJDK
rpm -qa|grep java
yum remove java*
1.9.2安装 jdk1.8
这里自行百度
2. 正式开始安装CDH
2.1 下载
2.1.1 下载cloudera-manager
Cloudera Manager下载地址：
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.1.3_x86_64.tar.gz
部署CDH的Parcel
2.1.2 下载CDH的Parcel文件
下载地址：http://archive.cloudera.com/cdh5/parcels/5.13.0/
CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel
CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha1
manifest.json
2.2 将下载的 Cloudera Manager放到指定位置
2.2.1 将下载的 Cloudera Manager文件放到指定位置
将 cloudera-manager-el6-cm5.13.0_x86_64.tar.gz 传输到所有节点的/opt目录下并解压到当前目录，
[root@hadoop01 opt]# tar -zxvf cloudera-manager-el6-cm5.13.0_x86_64.tar.gz
2.2.2.Agent配置
修改所有节点中/opt/cm-5.13.0/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名
[root@hadoop01 ~]#vim /opt/cm-5.15.0/etc/cloudera-scm-agent/config.ini
server_host=hadoop01
2..3 创建用户
2.3.1 在所有节点创建cloudera-scm用户
[root@hadoop01 ~]# useradd --system --home=/opt/cm-5.13.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
修改权限
[root@hadoop01 opt]# chown -R cloudera-scm:cloudera-scm cloudera
[root@hadoop02 opt]# chown -R cloudera-scm:cloudera-scm cm-5.13.0
2.4 准备Parcels，用以安装CDH5
2.4.1将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中
相关的文件如下：
CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel
CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha1
manifest.json
2.4.2 修改 CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha1命名
将CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha1，重命名为CDH-5.13.0-1.cdh5.13.0.p0.29-el6.parcel.sha,这点必须注意，否则，系统会重新下载CDH-5.1.3-1.cdh5.1.3.p0.12-el6.parcel文件
2.5 为Cloudera Manager 5建立数据库
2.5.1安装mysql驱动包
首先需要去MySql的官网下载JDBC驱动，http://dev.mysql.com/downloads/connector/j/，解压后，找到mysql-connector-java-5.1.33-bin.jar，放到/usr/share/java
2.5.2在主节点初始化CM5的数据库：
[root@hadoop01 ~]# /opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host localhost scm scm 123456
2.5.3启动
在主节点启动cloudera-scm-server：
[root@hadoop01 ~]#/opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
在所有节点启动cloudera-scm-agent：
[root@hadoop01 ~]#/opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start
检查端口是否监听：
[root@hadoop01 ~]# netstat -apn |grep 7180
3.集群安装
3.1 访问CM
通过http:// 172.23.216.73:7180/cmf/login访问CM
用户名：admin 密码：admin
3.2 为cdh 集群指定安装主机
3.3. 设置CDH的Parcel地址
3.4向集群所有主机安装并激活Parcel
3.5 检查主机正确性
3.6设置集群安装的服务组合
3.7自定义角色分配
3.8配置审核更改
3.9 集群首次运行
3.10集群安装成功
补充：
搭建中曾碰到的问题：
- cm注册到mysql的库和hive注册到MySQL的库不可以是同一个
- 拷贝parcel包到parcel-repo目录下时，manifest.json文件不要忘了拷贝，安装其他应用时，其他组件的parcel包如果也带有manifest.json，记得要将manifest.json合并。
- Hive的驱动包要放在/usr/share/java
- opt目录下相关文件的权限不要忘了修改
- spark安装时，jdk的目录需要放在/usr/java下
- phonex安装需要重新编译
- 删除cdh集群前需要将集群中的组件全部停止
- zookeeper安装时要选最好是奇数个节点，且一般五个节点就够了
- 给组件分配角色的时候最好将Cloudera Manager的监控角色和大数据角色分开
- spark和hive最好放一个节点，metastore不然无法同步
费劲了千辛万苦终于把公司的CDH大数据集群装好了，且能正常运行
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
