前期准备:
-
- JDK环境
版本:jdk-8u101-linux-x64.rpm
下载地址:oracle官网 - mysql
rpm包:http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.26-1.linux_glibc2.5.x86_64.rpm-bundle.tar
jdbc连接包mysql-connector-java-5.1.39-bin.jar:
http://dev.mysql.com/downloads/connector/j/ - CDH安装相关的包
- cloudera manager包 :5.7.2 cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz
下载地址:http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz - CDH-5.7.2-1.cdh5.7.2.p0.11-el7.parcel
- CDH-5.7.2-1.cdh5.7.2.p0.11-el7.parcel.sha1
- manifest.json
以上三个下载地址在http://archive.cloudera.com/cdh5/parcels/5.7.2/,注意centos要下载el7的,我就因为一开始不清楚下的el6,结果提示parcels不知道redhat7,搞了好久才还原到初始重新来过
- cloudera manager包 :5.7.2 cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz
- JDK环境
以上截图是我在此处安装过程中所用到的安装包
集群的规划
IP地址 | 主机名 | 说明 |
---|---|---|
192.168.160.130 | hadoop1 | 主节点master,datanode |
192.168.160.131 | hadoop2 | datanode |
192.168.160.132 | hadoop3
| datanode |
开始安装前配置和预装软件
- 安装jdk(每台机器都要)
首先卸载原有的openJDK
[root@hadoop1~]$ java -version java version "1.7.0_75" OpenJDK Runtime Environment (rhel-2.5.4.2.el7_0-x86_64 u75-b13) OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode) [root@hadoop1~]$ rpm -qa | grep jdk java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64 java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64 [root@hadoop1~]# yum -y remove java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64 [root@hadoop1~]# yum -y remove java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64 [root@hadoop1~]# java -version bash: /usr/bin/java: No such file or directory [root@hadoop1~]# rpm -ivh jdk-8u101-linux-x64.rpm [root@hadoop1~]# java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
- 修改每台节点服务器的有关配置hostname、selinux关闭,防火墙关闭
hostname修改:分别对三台都进行更改,并且注意每台名称和ip,每台都要配上
[root@hadoop1~]# vi /etc/sysconfig/network NETWORKING=yes HOSTNAME=hadoop1
[root@hadoop1~]# vi /etc/hosts 127.0.0.1 localhost.hadoop1 192.168.160.132 hadoop3 192.168.160.130 hadoop1 192.168.160.131 hadoop2
selinux关闭(所有节点官方文档要求)
[root@hadoop1~]# vim /etc/sysconfig/selinux SELINUX=disabled 重启才能生效 重启后检查 [root@hadoop1~]#sestatus -v SELinux status: disabled 表示已经关闭了
关闭防火墙(注意centos 6 是iptables)
[root@hadoop1~]# systemctl stop firewalld [root@hadoop1~]# systemctl disable firewalld rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service' rm '/etc/systemd/system/basic.target.wants/firewalld.service' [root@hadoop1~]# systemctl status firewalld firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled) Active: inactive (dead)
NTP服务器设置(用于不同节点间实现时间同步,详细的配置教程在我的另一边博客里面有写)
[root@hadoop1~]# systemctl stop firewalld [root@hadoop1~]# systemctl disable firewalld rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service' rm '/etc/systemd/system/basic.target.wants/firewalld.service' [root@hadoop1~]# systemctl status firewalld firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled) Active: inactive (dead)
- SSH无密码登录配置
安装过程中master需要各个节点的root免登录密码
先在master上生成公钥
[root@hadoop1]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): [enter] Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): [enter] Enter same passphrase again: [enter] Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 1d:b1:99:51:31:d8:f6:6c:b1:84:f9:af:7b:2c:72:dd root@hadoop1 The key's randomart image is: +--[ RSA 2048]----+ | o+++ | | .*=.o | | =. = o | | . . * | | S . . . | | .| | +.| | . + E| | o.+ | +-----------------+ [root@hadoop1]# ssh-copy-id 192.168.160.131 The authenticity of host '192.168.160.131 (192.168.160.131)' can't be established. ECDSA key fingerprint is e6:81:3f:9e:e6:bb:43:36:c7:4e:0f:6f:63:b2:12:a0. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@192.168.160.131's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '192.168.160.131'" and check to make sure that only the key(s) you wanted were added.
分别对两台节点进行以上操作,操作完了可以通过ssh ip进行验证是否可以无需输入密码就能直接登录到节点服务器上去
-
- 安装mysql
centos7自带的是mariadb,需要先卸载掉
- 安装mysql
[root@hadoop1]# rpm -qa | grep mariadb mariadb-libs-5.5.41-2.el7_0.x86_64 [root@hadoop1 huxin]# rpm -e --nodeps mariadb-libs-5.5.41-2.el7_0.x86_64
将下载好的MySQL rpm包拷贝到服务器上然后解压,安装
[root@hadoop1]# tar -xvf MySQL-5.6.24-1.linux_glibc2.5.x86_64.rpm-bundle.tar [root@hadoop1]#rpm -ivh MySQL-*.rpm
以上就是安装好了。然后就初始化mysql
然后初始化数据库执行
[root@hadoop1]#/usr/bin/mysql_install_db
然后就可以执行上面的初始化语句了
- 启动mysql [root@hadoop1]# service mysql restart ERROR! MySQL server PID file could not be found! Starting MySQL... SUCCESS! - 查看mysql root初始化密码 [root@hadoop1]# cat /root/.mysql_secret # The random password set for the root user at Fri Sep 16 11:13:25 2016 (local time): 9mp7uYFmgt6drdq3 - 登录进行去更改密码 [root@hadoop1]# mysql -u root -p mysql> SET PASSWORD=PASSWORD('123456'); - 允许mysql远程访问 mysql> update user set host='%' where user='root' and host='localhost'; Query OK, 1 row affected (0.05 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) - 配置开机启动 [root@hadoop1]# chkconfig mysql on
第三方依赖包
所有节点都安装
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi
注意这个地方依赖包不安装完下面启动集群的时候会死活启动不了的,这是血的教训啊!
在hadoop1上准备mysql的jar包
[root@hadoop1]# mkdir -p /usr/share/java
修改jar包的名字,并拷贝到/usr/share/java/目录(下面会有截图说明为什么修改)
[root@hadoop1]# cp mysql-connector-java-5.1.36-bin.jar /usr/share/java/mysql-connector-java.jar
安装Cloudera-Manager
- 解压cm tar包到指定目录,所有服务器都要
[root@hadoop1 ~]#mkdir /opt/cloudera-manager [root@hadoop1 ~]# tar -axvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz -C /opt/cloudera-manager
- 创建cloudera-scm用户(所有节点)
[root@hadoop1 ~]# useradd -r -d /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm [root@hadoop1 ~]# id cloudera-scm
- 配置从节点cloudera-manger-agent指向主节点服务器(根据集群的分配这里面应当全部节点都需要)
vim /opt/cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/config.ini 将server_host改为CMS所在的主机名即hadoop1
- 主节点中创建parcel-repo仓库目录(这个只在主节点hadoop1上需要)
[root@hadoop1 ~]# mkdir -p /opt/cloudera/parcel-repo [root@hadoop1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo [root@hadoop1 ~]# cp CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo 注意:其中CDH-5.7.2-1.cdh5.7.2.p0.18-el5.parcel.sha1 后缀要把1去掉
- 所有节点创建parcels目录(所有节点)
[root@hadoop1 ~]# mkdir -p /opt/cloudera/parcels [root@hadoop1 ~]# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels 解释:Clouder-Manager将CDHs从主节点的/opt/cloudera/parcel-repo目录中抽取出来,分发解压激活到各个节点的/opt/cloudera/parcels目录中
- 初始脚本配置数据库scm_prepare_database.sh(在主节点上)
[root@hadoop1 ~]# /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/scm_prepare_database.sh mysql -hhadoop1 -uroot -p123456 --scm-host hadoop1 scmdbn scmdbu scmdbp
说明:这个脚本就是用来创建和配置CMS需要的数据库的脚本。各参数是指:
mysql:数据库用的是mysql,如果安装过程中用的oracle,那么该参数就应该改为oracle。
-hhadoop1:数据库建立在hadoop1主机上面。也就是主节点上面。
-uroot:root身份运行mysql。-123456:mysql的root密码是***。
--scm-host hadoop1:CMS的主机,一般是和mysql安装的主机是在同一个主机上。 最后三个参数是:数据库名,数据库用户名,数据库密码。
在这个地方就可以解释上面为什么要改jar名了
- 启动主节点cloudera-scm-server
[root@hadoop1 ~]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
[root@hadoop1 ~]#chkconfig cloudera-scm-server on 同时为了保证在每次服务器重启的时候都能启动cloudera-scm-server,应该在开机启动脚本/etc/rc.local中加入命令:service cloudera-scm-server restart
- 启动cloudera-scm-agent所有节点
[root@hadoopX ~]# mkdir /opt/cloudera-manager/cm-5.7.2/run/cloudera-scm-agent
[root@hadoopX ~]# cp /opt/cloudera-manager/cm-5.7.2/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
[root@hadoopX ~]# chkconfig cloudera-scm-agent on
同时为了保证在每次服务器重启的时候都能启动cloudera-scm-agent,应该在开机启动脚本/etc/rc.local中加入命令:service cloudera-scm-agent restart
等待主节点安装并且启动就在浏览器中进行操作了
进入192168.160.130:7180 默认使用admin admin登录
此时已经完成一半的工作量了,出现这个界面说明CM已经安装成功了,下面就在这个web界面中部署CDH吧!
选择express版本
组件提示
配置主机!由于我们在各个节点都安装启动了agent,并且在中各个节点都在配置文件中指向hadoop1是server节点,所以各个节点的agent就会给agent发消息报告,所以这里我们可以在“当前管理的主机”中看到三个主机,全部勾选并继续,注意如果cloudera-scm-agent没有设为开机启动,如果以上有重启这里可能会检测不到其他服务器。
选择cdh
分发parcels到各个节点
主机配置正确性的检测
这个地方要注意这个地方有两项没有检查通过,可以在集群中使用以下命令
echo 0 > /proc/sys/vm/swappiness echo never > /sys/kernel/mm/transparent_hugepage/defrag
然后再点击上面的重新运行会发现这次全部检查通过了
选择要安装的服务,这里选择所有服务
角色分配
数据库设置选择
集群审核,这里都默认的
开始安装
安装完成
这个时候安装完成了,可以在浏览器中进入192.168.160.130:7180地址,查看集群情况,我这里有挺多报警,大概查看下基本都是内存或者存储空间使用阈值的报警,由于我们是本地虚拟机的,所以这些条件都有限,这里暂时不care这些报警了
安装完成!!!