官方指导手册:
https://www.cloudera.com/documentation/enterprise/latest/topics/introduction.html
一、准备工作,系统环境搭建
预先准备工作:
CDH 系统优化之关闭透明大页面及设置swappiness
http://blog.csdn.net/post_yuan/article/details/54967508
1 关闭透明大页面
首先查看透明大页是否启用,[always] never表示已启用,always [never]表示已禁用
[root@n12 ~]# cat /sys/kernel/mm/redhat_transparent_hugepage/defrag [always] madvise never
- 1
- 2
如果是启用状态,修改/etc/rc.local文件并添加以下两行,
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
上述修改要重启才能生效,如果希望暂时生效,可以直接运行上面两个命令。
2 修改swappiness
Linux内核参数vm.swappiness,值的范围为0~100,表示系统什么时候开始进行物理内存与虚拟内存的交换。举个例子,系统总内存为64G,vm.swappiness为60,表示在系统内存使用64*0.4=25.6G的时候开始物理内存与虚拟内存的交换,这个动作势必会影响系统的性能。因此,Cloudera建议把这个值修改为1~10。
首先检查当前设置的vm.swappiness值,
[root@n12 ~]# cat /proc/sys/vm/swappiness 60
- 1
- 2
临时修改swappiness(重启后恢复)
[root@n12 ~]# sysctl -w vm.swappiness=10 vm.swappiness = 10
- 1
- 2
永久生效(重启后不恢复)
echo "vm.swappiness=10" >> /etc/sysctl.conf
1、创建data目录,挂载磁盘
[root@cdh5node1 ~]# ls /data/ -al
total 4
drwxr-xr-x. 4 root root 34 Dec 20 10:07 .
dr-xr-xr-x. 18 root root 4096 Dec 20 10:14 ..
drwxr-xr-x 2 root root 6 Sep 21 11:15 disk01
drwxr-xr-x 2 root root 6 Sep 21 11:18 disk02
[root@cdh5node1 ~]# df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-root 3.7T 4.7G 3.7T 1% /
devtmpfs 126G 0 126G 0% /dev
tmpfs 126G 84K 126G 1% /dev/shm
tmpfs 126G 9.6M 126G 1% /run
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/sdb1 3.7T 34M 3.7T 1% /data/disk01
/dev/sdc1 3.7T 34M 3.7T 1% /data/disk02
/dev/sda2 1016M 167M 850M 17% /boot
/dev/sda1 200M 9.5M 191M 5% /boot/efi
tmpfs 26G 12K 26G 1% /run/user/42
tmpfs 26G 0 26G 0% /run/user/0
2、修改hostname,并配置/etc/hosts 文件
使用nmtui命令修改hostname
[root@cdh5node1 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.18.218.17 cdh5node1
10.18.218.5 cdh5node2
10.18.218.9 cdh5node3
3、配置主节点ssh免密码登录
[root@cdh5node1 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
99:60:26:05:37:b7:f4:13:67:86:a2:8e:b5:71:e9:98 root@cdh5node1
The key's randomart image is:
+--[ RSA 2048]----+
| ..+ o ..+ |
| o +.o.= |
| . +..oo |
| ++.oo . |
| + BS |
| . E . |
| |
| |
| |
+-----------------+
[root@cdh5node1 ~]# scp /root/.ssh/id_rsa.pub root@cdh5node2:/root/.ssh/
The authenticity of host 'cdh5node2 (10.18.218.5)' can't be established.
ECDSA key fingerprint is 7d:9a:d2:1b:2e:39:3c:26:94:55:1b:15:7a:fa:63:0e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cdh5node2,10.18.218.5' (ECDSA) to the list of known hosts.
root@cdh5node2's password:
id_rsa.pub 100% 396 0.4KB/s 00:00
[root@cdh5node1 ~]# scp /root/.ssh/id_rsa.pub root@cdh5node3:/root/.ssh/
The authenticity of host 'cdh5node3 (10.18.218.9)' can't be established.
ECDSA key fingerprint is 60:14:1f:42:fb:49:ba:1c:1b:a8:33:03:5c:45:81:51.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cdh5node3,10.18.218.9' (ECDSA) to the list of known hosts.
root@cdh5node3's password:
id_rsa.pub 100% 396 0.4KB/s 00:00
[root@cdh5node1 ~]# ssh root@cdh5node2
Last login: Wed Dec 20 10:19:22 2017 from 10.18.218.87
[root@cdh5node2 ~]# exit
logout
Connection to cdh5node2 closed.
在每台slave机器上执行
[root@cdh5node2 .ssh]# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
[root@cdh5node2 .ssh]# chmod 600 /root/.ssh/authorized_keys
4、安装jdk
首先删除CentOS自带的Open JDK
[root@cdh5node2 yum.repos.d]# yum remove java
java-1.7.0-openjdk-headless.x86_64 java-1.8.0-openjdk-headless.x86_64 javapackages-tools.noarch
java-1.7.0-openjdk.x86_64 java-1.8.0-openjdk.x86_64
[root@cdh5node2 yum.repos.d]# yum remove java-1.7.0-openjdk-headless.x86_64 java-1.8.0-openjdk-headless.x86_64 javapackages-tools.noarch java-1.7.0-openjdk.x86_64 java-1.8.0-openjdk.x86_64
Loaded plugins: fastestmirror, langpacks
Resolving Dependencies
--> Running transaction check
---> Package java-1.7.0-openjdk.x86_64 1:1.7.0.111-2.6.7.8.el7 will be erased
--> Processing Dependency: java >= 1.5 for package: jline-1.0-8.el7.noarch
---> Package java-1.7.0-openjdk-headless.x86_64 1:1.7.0.111-2.6.7.8.el7 will be erased
--> Processing Dependency: java-headless >= 1:1.6 for package: 1:libreoffice-core-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libjvm.so()(64bit) for package: 1:libreoffice-ure-5.0.6.2-3.el7.x86_64
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.102-4.b14.el7 will be erased
--> Processing Dependency: java-1.8.0-openjdk for package: icedtea-web-1.6.2-4.el7.x86_64
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.102-4.b14.el7 will be erased
---> Package javapackages-tools.noarch 0:3.4.1-11.el7 will be erased
--> Processing Dependency: jpackage-utils for package: rhino-1.7R4-5.el7.noarch
--> Processing Dependency: jpackage-utils for package: rhino-1.7R4-5.el7.noarch
--> Processing Dependency: jpackage-utils for package: tagsoup-1.2.1-8.el7.noarch
--> Processing Dependency: jpackage-utils >= 1.6 for package: tagsoup-1.2.1-8.el7.noarch
--> Running transaction check
---> Package icedtea-web.x86_64 0:1.6.2-4.el7 will be erased
---> Package jline.noarch 0:1.0-8.el7 will be erased
---> Package libreoffice-core.x86_64 1:5.0.6.2-3.el7 will be erased
--> Processing Dependency: libavmedialo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libclewlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libdbtoolslo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libdbtoolslo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libdrawinglayerlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libfilelo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nutil.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nutil.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libmsfilterlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libmsfilterlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libooxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libooxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libopencllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-draw-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-pyuno-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core for package: unoconv-0.6-7.el7.noarch
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-langpack-en-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsblo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsblo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-w