linux 安装 cdh5 详细步骤

官方指导手册：

https://www.cloudera.com/documentation/enterprise/latest/topics/introduction.html

 

一、准备工作，系统环境搭建

预先准备工作：

 

CDH 系统优化之关闭透明大页面及设置swappiness

http://blog.csdn.net/post_yuan/article/details/54967508

1 关闭透明大页面

首先查看透明大页是否启用，[always] never表示已启用，always [never]表示已禁用

[root@n12 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/defrag [always] madvise never

• 1
• 2

如果是启用状态，修改/etc/rc.local文件并添加以下两行，

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

上述修改要重启才能生效，如果希望暂时生效，可以直接运行上面两个命令。

2 修改swappiness

Linux内核参数vm.swappiness，值的范围为0~100，表示系统什么时候开始进行物理内存与虚拟内存的交换。举个例子，系统总内存为64G，vm.swappiness为60，表示在系统内存使用64*0.4=25.6G的时候开始物理内存与虚拟内存的交换，这个动作势必会影响系统的性能。因此，Cloudera建议把这个值修改为1~10。

首先检查当前设置的vm.swappiness值，

[root@n12 ~]# cat /proc/sys/vm/swappiness 60

• 1
• 2

临时修改swappiness（重启后恢复）

[root@n12 ~]# sysctl -w vm.swappiness=10 vm.swappiness = 10

• 1
• 2

永久生效（重启后不恢复）

echo "vm.swappiness=10" >> /etc/sysctl.conf

 

1、创建data目录，挂载磁盘

[root@cdh5node1 ~]# ls /data/ -al

total 4

drwxr-xr-x. 4 root root 34 Dec 20 10:07 .

dr-xr-xr-x. 18 root root 4096 Dec 20 10:14 ..

drwxr-xr-x 2 root root 6 Sep 21 11:15 disk01

drwxr-xr-x 2 root root 6 Sep 21 11:18 disk02

[root@cdh5node1 ~]# df -hl

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/cl-root 3.7T 4.7G 3.7T 1% /

devtmpfs 126G 0 126G 0% /dev

tmpfs 126G 84K 126G 1% /dev/shm

tmpfs 126G 9.6M 126G 1% /run

tmpfs 126G 0 126G 0% /sys/fs/cgroup

/dev/sdb1 3.7T 34M 3.7T 1% /data/disk01

/dev/sdc1 3.7T 34M 3.7T 1% /data/disk02

/dev/sda2 1016M 167M 850M 17% /boot

/dev/sda1 200M 9.5M 191M 5% /boot/efi

tmpfs 26G 12K 26G 1% /run/user/42

tmpfs 26G 0 26G 0% /run/user/0

 

2、修改hostname，并配置/etc/hosts 文件

使用nmtui命令修改hostname

[root@cdh5node1 ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

 

10.18.218.17 cdh5node1

10.18.218.5 cdh5node2

10.18.218.9 cdh5node3

 

3、配置主节点ssh免密码登录

[root@cdh5node1 ~]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

99:60:26:05:37:b7:f4:13:67:86:a2:8e:b5:71:e9:98 root@cdh5node1

The key's randomart image is:

+--[ RSA 2048]----+

| ..+ o ..+ |

| o +.o.= |

| . +..oo |

| ++.oo . |

| + BS |

| . E . |

| |

| |

| |

+-----------------+

[root@cdh5node1 ~]# scp /root/.ssh/id_rsa.pub root@cdh5node2:/root/.ssh/

The authenticity of host 'cdh5node2 (10.18.218.5)' can't be established.

ECDSA key fingerprint is 7d:9a:d2:1b:2e:39:3c:26:94:55:1b:15:7a:fa:63:0e.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'cdh5node2,10.18.218.5' (ECDSA) to the list of known hosts.

root@cdh5node2's password:

id_rsa.pub 100% 396 0.4KB/s 00:00

[root@cdh5node1 ~]# scp /root/.ssh/id_rsa.pub root@cdh5node3:/root/.ssh/

The authenticity of host 'cdh5node3 (10.18.218.9)' can't be established.

ECDSA key fingerprint is 60:14:1f:42:fb:49:ba:1c:1b:a8:33:03:5c:45:81:51.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'cdh5node3,10.18.218.9' (ECDSA) to the list of known hosts.

root@cdh5node3's password:

id_rsa.pub 100% 396 0.4KB/s 00:00

[root@cdh5node1 ~]# ssh root@cdh5node2

Last login: Wed Dec 20 10:19:22 2017 from 10.18.218.87

[root@cdh5node2 ~]# exit

logout

Connection to cdh5node2 closed.

 

在每台slave机器上执行

[root@cdh5node2 .ssh]# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

[root@cdh5node2 .ssh]# chmod 600 /root/.ssh/authorized_keys

 

4、安装jdk

首先删除CentOS自带的Open JDK

[root@cdh5node2 yum.repos.d]# yum remove java

java-1.7.0-openjdk-headless.x86_64 java-1.8.0-openjdk-headless.x86_64 javapackages-tools.noarch

java-1.7.0-openjdk.x86_64 java-1.8.0-openjdk.x86_64

[root@cdh5node2 yum.repos.d]# yum remove java-1.7.0-openjdk-headless.x86_64 java-1.8.0-openjdk-headless.x86_64 javapackages-tools.noarch java-1.7.0-openjdk.x86_64 java-1.8.0-openjdk.x86_64

Loaded plugins: fastestmirror, langpacks

Resolving Dependencies

--> Running transaction check

---> Package java-1.7.0-openjdk.x86_64 1:1.7.0.111-2.6.7.8.el7 will be erased

--> Processing Dependency: java >= 1.5 for package: jline-1.0-8.el7.noarch

---> Package java-1.7.0-openjdk-headless.x86_64 1:1.7.0.111-2.6.7.8.el7 will be erased

--> Processing Dependency: java-headless >= 1:1.6 for package: 1:libreoffice-core-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libjvm.so()(64bit) for package: 1:libreoffice-ure-5.0.6.2-3.el7.x86_64

---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.102-4.b14.el7 will be erased

--> Processing Dependency: java-1.8.0-openjdk for package: icedtea-web-1.6.2-4.el7.x86_64

---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.102-4.b14.el7 will be erased

---> Package javapackages-tools.noarch 0:3.4.1-11.el7 will be erased

--> Processing Dependency: jpackage-utils for package: rhino-1.7R4-5.el7.noarch

--> Processing Dependency: jpackage-utils for package: rhino-1.7R4-5.el7.noarch

--> Processing Dependency: jpackage-utils for package: tagsoup-1.2.1-8.el7.noarch

--> Processing Dependency: jpackage-utils >= 1.6 for package: tagsoup-1.2.1-8.el7.noarch

--> Running transaction check

---> Package icedtea-web.x86_64 0:1.6.2-4.el7 will be erased

---> Package jline.noarch 0:1.0-8.el7 will be erased

---> Package libreoffice-core.x86_64 1:5.0.6.2-3.el7 will be erased

--> Processing Dependency: libavmedialo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libclewlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libdbtoolslo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libdbtoolslo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libdrawinglayerlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libfilelo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libi18nutil.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libi18nutil.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libmsfilterlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libmsfilterlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libooxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libooxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libopencllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-draw-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-pyuno-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libreoffice-core for package: unoconv-0.6-7.el7.noarch

--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-langpack-en-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsblo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsblo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libsvxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libswlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libtklo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64

--> Processing Dependency: libtklo.so()(64bit) for package: 1:libreoffice-graphicfilter-5