CHD 5.10 离线安装

安装虚拟机，我用的是centos6.5的版本。

选择回车

重启机器

设置虚拟机网络

1.集群节点规划,安装三台机器，每台机器都执行以下操作，切记！

[root@hadoop-cdh1 ~]# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop-01  # 机器名称
GATEWAY=192.168.137.2  # 网关

[root@hadoop-cdh1 ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.137.130 hadoop-cdh1
192.168.137.131 hadoop-cdh2
192.168.137.132 hadoop-cdh3

[root@hadoop-cdh1 ~]# vi /etc/resolv.conf  
search localdomain
nameserver 192.168.137.2
nameserver 8.8.8.8                #ping 外网
nameserver 8.8.4.4

[root@hadoop-cdh1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.137.130    #ip
NETMASK=225.225.225.0

GATEWAY=192.168.137.2

修改网卡 如果是克隆的机器就要设置，删除原来的，保存克隆以后的
/etc/udev/rules.d/70-persistent-net.rules  

1) Linux操作系统中永久性生效，重启后不会复原
开启： chkconfig iptables on
关闭： chkconfig iptables off
2) 即时生效，重启后复原
开启： service iptables start
关闭： service iptables stop
service network restart 重启网卡
service iptables status可以查看到iptables服务的当前状态。
yum install -y lrzsz
yum -y install wget

yum -y install vim

三台机器设置完成后，reboot重启三台机器，检查机器配置。

hostname 是否正确。

hostname -i是否正确。

ping www .baidu.com是否可以ping通。

安装mysql

https://blog.csdn.net/nszkadrgg/article/details/78666628

配置hadoop用户的ssh信任关系(三台机器都要安装)

安装无密码登录(三台机器都要安装)

[root@hadoop-cdh1 ~]# yum -y install openssh-clients

[root@hadoop-cdh1 java]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/root/.ssh'.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
02:97:cf:e5:85:f4:3d:77:4b:24:18:23:9d:47:0e:1b root@hadoop-cdh1
The key's randomart image is:
+--[ RSA 2048]----+
|         .oE=o . |
|       . ..**oo  |
|    . o   o.+.o.o|
|     o o o .  .oo|
|      . S .    . |
|       .         |
|                 |
|                 |
|                 |

+-----------------+

[root@hadoop-cdh1 java]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[root@hadoop-cdh1 java]# chmod 0600 ~/.ssh/authorized_keys

测试三台机器ssh登录(每台都要测试)

[root@hadoop-cdh1 java]# ssh hadoop-cdh2
The authenticity of host 'hadoop-cdh2 (192.168.137.131)' can't be established.
RSA key fingerprint is 69:99:33:84:fe:91:f6:43:df:4b:0c:63:ba:9b:c1:fc.
Are you sure you want to continue connecting (yes/no)?
Host key verification failed.
[root@hadoop-cdh1 java]# ssh hadoop-cdh2
The authenticity of host 'hadoop-cdh2 (192.168.137.131)' can't be established.
RSA key fingerprint is 69:99:33:84:fe:91:f6:43:df:4b:0c:63:ba:9b:c1:fc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop-cdh2,192.168.137.131' (RSA) to the list of known hosts.
root@hadoop-cdh2's password:

Last login: Sat Apr 21 03:54:12 2018 from hadoop-cdh3

检查系统版本是否定一致 (每台都要做)

[root@hadoop-cdh2 ~]# cat /etc/redhat-release

CentOS release 6.5 (Final)

设置时钟同步(每台都要做)

[root@hadoop-cdh2 ~]# grep ZONE /etc/sysconfig/clock

ZONE="Asia/Shanghai"

修改机器的时间

[root@hadoop-cdh1 ~]# date
Sat Apr 21 04:07:25 CST 2018
[root@hadoop-cdh1 ~]# date -s 04/20/2018
Fri Apr 20 00:00:00 CST 2018
[root@hadoop-cdh1 ~]# date -s 20:32:00
Fri Apr 20 20:32:00 CST 2018
[root@hadoop-cdh1 ~]# dare
-bash: dare: command not found
[root@hadoop-cdh1 ~]# date
Fri Apr 20 20:32:06 CST 2018

安装ntpdate(每台机器都要安装)

[root@hadoop-cdh3 ~]# yum -y install ntpdate
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.zju.edu.cn
 * extras: mirrors.aliyun.com
 * updates: mirrors.njupt.edu.cn
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package ntpdate.x86_64 0:4.2.6p5-12.el6.centos.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================================================================================================
 Package                                         Arch                                           Version                                                          Repository                                       Size
=======================================================================================================================================================================================================================
Installing:
 ntpdate                                         x86_64                                         4.2.6p5-12.el6.centos.2                                          updates                                          79 k

Transaction Summary
=======================================================================================================================================================================================================================
Install       1 Package(s)

Total download size: 79 k
Installed size: 123 k
Downloading Packages:
ntpdate-4.2.6p5-12.el6.centos.2.x86_64.rpm                                                                                                                                                      |  79 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : ntpdate-4.2.6p5-12.el6.centos.2.x86_64                                                                                                                                                              1/1
  Verifying  : ntpdate-4.2.6p5-12.el6.centos.2.x86_64                                                                                                                                                              1/1

Installed:
  ntpdate.x86_64 0:4.2.6p5-12.el6.centos.2                                                                                                                                                                            

Complete!

安装ntp服务(三台都要安装)

yum -y install ntp

启动ntpd服务(三台机器都要启动)

[root@hadoop-cdh1 ~]# service ntpd start

Starting ntpd:                                             [  OK  ]

时间设置同步

[root@hadoop-cdh1 ~]# ntpdate hadoop-cdh2

20 Apr 20:41:06 ntpdate[2676]: no server suitable for synchronization found

[root@hadoop-cdh1 ~]# ntpdate hadoop-cdh3
20 Apr 20:41:06 ntpdate[2676]: no server suitable for synchronization found

安装和启动http服务

[root@hadoop-cdh1 ~]# rpm -qa|grep httpd
httpd-tools-2.2.15-60.el6.centos.6.x86_64
httpd-2.2.15-60.el6.centos.6.x86_64
[root@hadoop-cdh1 ~]#

其他两台没有安装，安装hadoop-cdh2，hadoop-cdh3两台的http服务，安装  yum install -y httpd

下面的三步三台都做

[root@hadoop-cdh1 ~]# chkconfig --list|grep httpd
httpd              0:off    1:off    2:on    3:on    4:on    5:on    6:off

[root@hadoop-cdh1 ~]# chkconfig httpd on

[root@hadoop-cdh1 ~]# chkconfig --list|grep httpd
httpd              0:off    1:off    2:on    3:on    4:on    5:on    6:off

[root@hadoop-cdh1 ~]# service httpd start

Starting httpd:

安装jdk，一定要用oracle版本的jdk，官网上可以下载，不要安装linux自带的openJDK。

[root@hadoop001 ~]# rz #上传jdk-8u45-linux-x64.gz
[root@hadoop-cdh1 java]# mkdir -p /usr/java
[root@hadoop-cdh1 java]# mv jdk-8u45-linux-x64.gz /usr/java

[root@hadoop-cdh1 java]# tar -xzvf jdk-8u45-linux-x64.gz

解压后的jdk复制到hadoop-cdh2，hadoop-cdh3

其他的两台创建mkdir -p /usr/java目录

scp到其他的两台机器

scp -r /usr/java/jdk1.8.0_45/ hadoop-cdh2:/usr/java

scp -r /usr/java/jdk1.8.0_45/ hadoop-cdh3:/usr/java

配置jdk的环境变量

[root@hadoop-cdh1 java]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45

export PATH=$JAVA_HOME/bin:$PATH

[root@hadoop-cdh1 java]# source /etc/profile

[root@hadoop-cdh1 java]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

scp 到其他的两台机器

[root@hadoop-cdh1 java]# scp -r /etc/profile/ hadoop-cdh2:/etc/profile

[root@hadoop-cdh1 java]# scp -r /etc/profile/ hadoop-cdh3:/etc/profile

其他的两台机器source /ect/profile

接下来安装CDH

http://archive.cloudera.com/cdh5/parcels/5.10.0/

到CDH官网下载对应版本的parcels文件

我们存在parcel的位置是 /var/www/html

[root@hadoop-cdh1 html]# mkdir parcels

[root@hadoop-cdh1 html]# cd parcels/
[root@hadoop-cdh1 parcels]# ll
total 1466572
-rw-r--r--. 1 root root 1501694035 Dec 10 19:14 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel
-rw-r--r--. 1 root root         41 Dec 10 18:51 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha
-rw-r--r--. 1 root root      64807 Dec 10 19:15 manifest.json
[root@hadoop-cdh1 parcels]# pwd
/var/www/html/parcels

CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1  后面的1去掉，之前我去掉了

[root@hadoop-cdh1 parcels]# mv CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1 CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha

校验你下载的文件是不是全的，如果不全也是不会安装成功的，所以一定要校验。

[root@hadoop-cdh1 parcels]# sha1sum CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel
52f95da433f203a05c2fd33eb0f144e6a5c9d558  CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel

[root@hadoop-cdh1 parcels]# cat CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha

52f95da433f203a05c2fd33eb0f144e6a5c9d558

下载cm

http://archive.cloudera.com/cm5/repo-as-tarball/5.10.0/

校验下载的文件

解压

[root@hadoop-cdh1 html]# tar -zxvf cm5.10.0-centos6.tar.gz

然后创建和官网一样的路径

修改文件名称

配置本地的yum源，集群自安装的时候就会从本地down包，不会从官网down(三台机器都要配置)

vim /etc/yum.repos.d/cloudrea-manager.repo   配置的内容一样，没有cloudrea-manager.repo文件就创建

在浏览器上查看是否配置成功，能打开说明配置成功。

官网参考链接

http://archive.cloudera.com/cdh5/parcels/5.10/

http://archive.cloudera.com/cm5/repo-as-tarball/5.10.0/

http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.10/

install server rpm in cm instance

install server

mysql创建元数据库和用户，两个都创建

修改service和mysql的配置

vim /etc/cloudera-scm-server/db.properties  修改后保存

添加mysql的驱动包

service 启动的日志

用户名密码默认admin

下一步

下一步

下一步

下一步

下一步

下一步

下一步

下一步

这一步很重要，点击下一步

开始分发到每台机器

下一步

关闭大页面

设置swap，三台机器都要设置

重新Run Again

没有黄色的提示了

现在安装hadoop的三个核心组件 ，hdfs, yarn,zookeeper

下面设置机器的角色分配

角色分配:
192.168.137.130  hadoop-cdh1
192.168.137.131  hadoop-cdh2
192.168.137.132  hadoop-cdh3

第一阶段角色:
hadoop-cdh1: mysql cm-server cm-agent
hadoop-cdh2: cm-agent

hadoop-cdh3: cm-agent

cm(5个服务):hadoop-cdh1

第二阶段角色:
hdfs 、yarn 、zk

hadoop-cdh1: NN  DN    RM NM  ZK
hadoop-cdh2: DN  SNN   NM     ZK
hadoop-cdh3: DN        NM     ZK

选择下一步，这里一定要有mysql 的驱动包，目录就按照下面的目录放置，别的目录有时候会找不到

[root@hadoop-cdh1 cloudera-manager]# cd /usr/share/java/
[root@hadoop-cdh1 java]# ll
total 964
-rw-r--r--. 1 root root 985603 Oct 22  2017 mysql-connector-java.jar
[root@hadoop-cdh1 java]#

点击下一步，所有的都不要变

点击下一步，安装服务

安装完成，点击完成。

设置namenode最小1G。

查找heap，点击菜单的namenode，设置为1G，点击保存。

返回主菜单

重启

下一步

点击Finish

点击Finish

其他的黄色和红色的信息，根据你自己集群的情况，如果资源充足就设置，不充足就抑制。

service cloudera-scm-agent status 查看agent的状态(每台)

service cloudera-scm-service status 查看server的状态

正确的关机顺序

service cloudera-scm-agent stop  先关闭agent

service cloudera-scm-service stop  再关闭service

service mysql  stop  再关闭mysql

开机顺序刚好相反，先启动mysql，再启动server，然后再启动agent。