安装cloudera hadoop

环境：centos 7.2 node136-node140 主节点：node136 从节点： node137-node140
软件获取 https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.14.2/RPMS/x86_64/
配置所有节点间免密码ssh登录
所有节点关闭selinux与防火墙
[root@node136 ~]# sed -i -e ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/sysconfig/selinux
[root@node136 ~]# sed -i -e ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/selinux/config
[root@node136 ~]# systemctl disable firewalld

在所有节点上开启ntpd服务，同步同一个时间服务器的时间
[root@node136 ~]# vim /etc/ntp.conf 配置为内网中ntp服务器的地址
[root@node136 ~]# systemctl start ntpd
[root@node136 ~]# systemctl enable ntpd
如果ntp服务器有问题，解决后需要重启cloudera-scm-server与cloudera-scm-agent
在所有节点上进行安装
[root@node136 ~]# yum -y install sysstat vim tree wget lrzsz screen gcc python-devel gcc-c++ vim ntpdate libyaml
libyaml-devel python-setuptools ntp fuse httpd mod_ssl openssl-devel python-psycopg2 MySQL-python redhat-lsb postgresql-server
[root@node136 ~]# cat setup_cloudera_env.sh

#!/bin/bash
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo    "* soft nofile 65536         " >>  /etc/security/limits.conf
echo    "* hard nofile 65536         " >>  /etc/security/limits.conf
echo    "root soft nofile 65536      " >>  /etc/security/limits.conf
echo    "root hard nofile 65536      " >>  /etc/security/limits.conf
echo    "* soft memlock unlimited    " >>  /etc/security/limits.conf
echo    "* hard memlock unlimited    " >>  /etc/security/limits.conf
echo    "root soft memlock unlimited " >>  /etc/security/limits.conf
echo    "root hard memlock unlimited " >>  /etc/security/limits.conf
echo    "* soft as unlimited         " >>  /etc/security/limits.conf
echo    "* hard as unlimited         " >>  /etc/security/limits.conf
echo    "root soft as unlimited      " >>  /etc/security/limits.conf
echo    "root hard as unlimited      " >>  /etc/security/limits.conf
echo "vm.max_map_count = 131072" >> /etc/sysctl.conf
echo "vm.swappiness=1" >> /etc/sysctl.conf
sysctl -p
rpm -ivh  /root/Hadoop/cloudera-manager-daemons-5.14.2-1.cm5142.p0.8.el7.x86_64.rpm
rpm -ivh  /root/Hadoop/cloudera-manager-server-5.14.2-1.cm5142.p0.8.el7.x86_64.rpm
rpm -ivh  /root/Hadoop/cloudera-manager-server-db-2-5.14.2-1.cm5142.p0.8.el7.x86_64.rpm
rpm -ivh  /root/Hadoop/enterprise-debuginfo-5.14.2-1.cm5142.p0.8.el7.x86_64.rpm
rpm -ivh  /root/Hadoop/cloudera-manager-agent-5.14.2-1.cm5142.p0.8.el7.x86_64.rpm

[root@node136 ~]# sh -x setup_cloudera_env.sh
可以只在agent节点安装cloudera-manager-agent与cloudera-manager-daemons的rpm，但这里对所有节点安装所有rpm包。
[root@node136 ~]# mkdir -p /opt/cloudera/parcel-repo

[root@node136 ~]# cp -rpv Hadoop/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel /opt/cloudera/parcel-repo/
“Hadoop/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel” -> “/opt/cloudera/parcel-repo/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel”
[root@node136 ~]# cp -rpv Hadoop/manifest.json /opt/cloudera/parcel-repo/
“Hadoop/manifest.json” -> "/opt/cloudera/parcel-repo/manifest.json
[root@node136 ~]# cp -rpv Hadoop/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha
“Hadoop/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1” -> “/opt/cloudera/parcel-repo/CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha”

[root@node136 ~]# yum -y erase java
[root@node136 ~]# rpm -ivh Hadoop/jdk-8u172-linux-x64.rpm
准备中… ################################# [100%]
正在升级/安装…
1:jdk1.8-2000:1.8.0_172-fcs ################################# [100%]
Unpacking JAR files…
tools.jar…
plugin.jar…
javaws.jar…
deploy.jar…
rt.jar…
jsse.jar…
charsets.jar…
localedata.jar…
[root@node137 ~]# echo “export JAVA_HOME=/usr/java/jdk1.8.0_172-amd64/” >> /etc/profile
[root@node137 ~]# source /etc/profile
[root@node136 ~]# ls /etc/init.d/|grep clouder
cloudera-scm-agent
cloudera-scm-server
cloudera-scm-server-db

配置好内网的yum源
[root@node136 ~]# cp Hadoop/cloudera-manager.repo /etc/yum.repos.d/cloudera-manager.repo
[root@node136 ~]# cat /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name=Cloudera Manager
baseurl=http://10.98.131.70/cm5/redhat/7/x86_64/cm/5/
gpgkey =http://10.98.131.70/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1
[root@node136 ~]# chmod +x ./Hadoop/cloudera-manager-installer.bin
[root@node136 ~]# ./Hadoop/cloudera-manager-installer.bin
在web页面上一步步安装就可以了。
在这里没有手动启动cloudera-scm-server与cloudera-scm-agent服务，有些文章说需要手动启动。
使用了内嵌postgres数据库，在登陆首页时会提示不是生产环境安装。

完成后查看状态如下：
主节点：

[root@node136 ~]# ps -ef|grep postgres
clouder+  4213  4714  0 4月27 ?       00:01:23 postgres: hive1 hive1 10.98.131.137(52180) idle
clouder+  4214  4714  0 4月27 ?       00:01:23 postgres: hive1 hive1 10.98.131.137(52181) idle
clouder+  4215  4714  0 4月27 ?       00:01:22 postgres: hive1 hive1 10.98.131.138(58389) idle
clouder+  4216  4714  0 4月27 ?       00:01:21 postgres: hive1 hive1 10.98.131.138(58390) idle
clouder+  4230  4714  0 4月27 ?       00:01:29 postgres: hive1 hive1 10.98.131.136(58431) idle
clouder+  4231  4714  0 4月27 ?       00:01:29 postgres: hive1 hive1 10.98.131.136(58432) idle
clouder+  4237  4714  0 4月27 ?       00:31:19 postgres: hive1 hive1 10.98.131.138(58393) idle
clouder+  4238  4714  0 4月27 ?       00:31:21 postgres: hive1 hive1 10.98.131.138(58394) idle
clouder+  4246  4714  0 4月27 ?       00:31:28 postgres: hive1 hive1 10.98.131.137(52188) idle
clouder+  4247  4714  0 4月27 ?       00:30:58 postgres: hive1 hive1 10.98.131.137(52189) idle
clouder+  4255  4714  0 4月27 ?       00:28:43 postgres: hive1 hive1 10.98.131.136(58440) idle
clouder+  4256  4714  0 4月27 ?       00:28:44 postgres: hive1 hive1 10.98.131.136(58441) idle
clouder+  4714     1  0 4月2