cloudera manager搭建hadoop环境入门

一、安装centos7虚拟机

https://blog.csdn.net/dongyuguoai/article/details/83988878

二、配置vmware网关和虚拟机网络

https://blog.csdn.net/dongyuguoai/article/details/83892530

https://blog.csdn.net/dongyuguoai/article/details/84035804

三、所有节点修改虚拟机主机名

https://blog.csdn.net/dongyuguoai/article/details/84035699

四、所有节点安装jdk8

https://www.cnblogs.com/coder-lzh/p/8746609.html

五、所有节点下载所需文件（CM包和CDH包）

http://archive.cloudera.com/cdh5/parcels/

http://archive.cloudera.com/cm5/cm/5/

http://archive.cloudera.com/cdh5/parcels/5.7/manifest.json

六、所有节点关闭防火墙和linux安全策略

systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
reboot

重启后查看状态：

getenforce
sestatus -v

七、所有节点安装时统服务器

yum -y install ntp

启动ntp服务：

systemctl start ntpd
systemctl enable ntpd

八、设置ssh无密码登录

主节点上：

生成秘钥：ssh-keygen -t rsa
配置免密节点：ssh-copy-id yourNodeIP

九、主节点上安装数据库

yum -y install mariadb-server
vi /etc/my.cnf
...
systemctl start mariadb 
systemctl enable mariadb
mysql_secure_installation

...

十、所有节点安装其他工具：

yum  -y  install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security spax time zlib-devel

chmod +x /etc/rc.d/rc.local;

echo "echo 10 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local;

echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local;

（xshell可能echo写不进去，到虚拟机本身的终端上却可以写进去，why？）

echo 10 > /proc/sys/vm/swappiness;
echo never > /sys/kernel/mm/transparent_hugepage/defrag; #禁用hugepage

echo never > /sys/kernel/mm/transparent_hugepage/enable; #禁用hugepage

yum -y install rpcbind;

systemctl start rpcbind;
echo "systemctl start rpcbind" >> /etc/rc.d/rc.local;

swappiness设置原则：

https://blog.csdn.net/wulantian/article/details/36184943

十一、主节点准备文件、解压文件、拷贝jar包

1. 所需文件：

CDH-***-***.parcel   cloudera-manager-***.tar.gz    manifest.json    CDH-***-***.parcel.sha1  mysql-connector-java.jar

wget http://archive.cloudera.com/cdh5/parcels/5.13.3/manifest.json

2. 解压文件：

tar -zxvf cloudera-manager-***.tar.gz -C /opt/

3. 拷jar包

cp mysql-connector-java.jar /opt/cm-5.13.3/share/cmf/lib/mysql-connector-java.jar

4.所有节点

创建/usr/share/java目录，将mysql-connector包再拷一份进去

十二、主节点制作本地源

cp CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/

cd /opt/cloudera/parcel-repo/

mv CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha1 CDH-5.13.3-1.cdh5.13.3.p0.2-el7.parcel.sha

十三、主节点更改配置：

sed -i "s/server_host=localhost/server_host=192.168.1.113/" /opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini

mkdir /opt/cm-5.13.3/run/cloudera-scm-agent

十四、除主节点外其他节点拷贝文件

scp -r 192.168.1.113:/opt/cm-5.13.3 /opt

十五、所有节点创建用户：

useradd --system --home=/opt/cm-5.13.3/run/cloudera-scm-server  --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

或者（复制下面的代码可能不行，手敲就可以，蛮奇怪的）：

useradd --system cloudera-scm

usermod --home=/opt/cm-5.13.3/run/cloudera-scm-server cloudera-scm

usermod --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

十六、主节点创建数据库：

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

grant all on *.* to root@"%" Identified by "123456";

十七、主节点测试数据库：

/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host localhost scm scm scm
 

十八、启动服务器和客户端

1. 主节点：

/opt/cm-5.13.3/etc/init.d/cloudera-scm-server start

2. 所有节点：

/opt/cm-5.13.3/etc/init.d/cloudera-scm-agent start

3. 查看日志是否有错

tail -f /opt/cm-5.13.3/log/cloudera-scm-server/cloudera-scm-server.log

十九、登录页面

 

 

注意：

https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html

1. jdk版本选择（本人jdk7安装，host monitor起不来，hue数据库连不上）

CDK 3.0 and higher Powered By Apache Kafka require JDK 8, and do not support JDK 7.
CDS 2.2 Powered By Apache Spark, which can be installed on CDH 5, requires JDK 8.

2. 内存问题，为了避免系统将cloudera server杀掉。（本文将Xmx调到小于2GB后，提示无法安装。所以将虚拟机内存调到4G）

However, to run the Cloudera Manager Server on a machine with 2 GB of RAM, you must tune down its maximum heap size (by modifying -Xmx in /etc/default/cloudera-scm-server). Otherwise the kernel might kill the Server for consuming too much RAM.

3. 其他方法安装：

原生Hadoop（及CDH5）：

文件下载：百度所需要的组件，到官网里downloads里找到清华镜像进行下载

hadoop生态系统：https://blog.csdn.net/qq_35038153/article/details/77824919

单机版原生hadoop+hive：https://www.cnblogs.com/xuwujing/p/8045821.html

https://blog.csdn.net/qazwsxpcm/article/details/78637874

CDH5安装5个节点的Hadoop集群：https://blog.csdn.net/u010270403/article/details/51446674

4. 本文参考的CM安装教程：

https://blog.csdn.net/oschina_41140683/article/details/81211635

https://www.cnblogs.com/fujiangong/p/5620050.html

https://www.cnblogs.com/haozhengfei/p/d90e8f4da465036fabbb1d1e1eae886a.html

 

错误解决：

https://blog.csdn.net/shushugood/article/details/76686062