Centos7 使用yum安装CDH5.14

概述

目标是搭建一个CDH 的测试环境
系统配置如下:

  • centos7.3 16G内存,200G磁盘(master节点)
  • centos7.3 16G内存,200G磁盘(node节点)
  • centos7.3 16G内存,200G磁盘(node节点)
  • centos7.3 16G内存,200G磁盘(node节点)
    搭建的cdh为最新的CDH5.14,感觉master内存还是太少,有点吃力
主机配置

192.168.0.1    cdh1
192.168.0.2    cdh2
192.168.0.3    cdh3
192.168.0.4    cdh4

搭建步骤

CDH的搭建有多种方法,一般来说,都是先搭建Cloudera Manager,然后利用Cloudera Manager搭建CDH,如果是测试环境,可以直接利用Cloudera Manager自动化安装,这种方式使用内嵌的PostgreSQL作为metadata等数据的存储,不适于生产环境。生产环境中一般会使用Mysql或其他独立搭建的数据库。

准备工作

在所有节点上都要执行的工作,包括host设置,设置yum repo,无密钥登录,ntp设置,安装jdk,关闭防火墙等等。

  • 修改 hostname,执行命令 hostname NAME,并修改 /etc/hostname文件。
  • 修改/etc/hosts文件

  • 192.168.0.1    cdh1
    192.168.0.2    cdh2
    192.168.0.3    cdh3
    192.168.0.4    cdh4
  • 其中cdh1为master节点,接着为每台主机配置hostname(cloudera-scm-agent需要)

  • $ hostname cdhxxx
    修改 /etc/sysconfig/network 配置文件,以便下次重启的时,使用新的主机名
    HOSTNAME=cdhxxx
    cdhxxx跟host所在地一致,比如192.168.0.1就是cdh1

  • 关闭防火墙.

  • $ systemctl stop firewalld
    $ systemctl disable firewalld
  • 所有节点设置无密码登陆

  • & yum -y install openssh-clients
    $ ssh-keygen -t rsa    #一直回车
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub cdh2
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub cdh3
    $ ssh-copy-id -i ~/.ssh/id_rsa.pub cdh4
    $ ssh root@cdh2    #测试
    注意:以上所有节点都需要配置
  • 设置 yum repo
  • $ curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
    $ curl -o /etc/yum.repos.d/cloudera-manager.repo https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
  • ntp设置,安装ntp,编辑crontab定时同步

    $ yum -y install ntp
    $ cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
    $ ntpdate time1.aliyun.com
    $ crontab -e


  • 输入:  30 02 * * *  ntpdate time1.aliyun.com
  • 安装jdk
  • yum install oracle-j2sdk1.7 -y
  • 关闭SElinux
  • # To disable SELinux until next reboot:
    setenforce Permissive
  • 修改/etc/selinux/config为disabled,重启生效。
    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #       enforcing - SELinux security policy is enforced.
    #       permissive - SELinux prints warnings instead of enforcing.
    #       disabled - SELinux is fully disabled.
    SELINUX=disabled
    # SELINUXTYPE= type of policy in use. Possible values are:
    #       targeted - Only targeted network daemons are protected.
    #       strict - Full SELinux protection.
    SELINUXTYPE=targeted

  • master节点安装与配置Mysql
    $ wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
    $ sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
    $ yum update -y
    $ sudo yum install -y mysql-server
    $ sudo systemctl start mysqld
  • 删除/var/lib/mysql/ib_logfile0 和 /var/lib/mysql/ib_logfile1文件,并配置/etc/my.conf 
    如:my.conf文件

    安装 MySQL JDBC Driver (mysql-connector-java-5.1.44-bin.jar网上找)
    $ mkdir -p /usr/share/java/
    $ cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/mysql-connector-java.jar
    $ sudo systemctl restart mysqld

    创建cdh对应的数据库。
    mysql -u root -p
    mysql> create database amon DEFAULT CHARACTER SET utf8;   #Activity Monitor
    mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password';

    mysql> create database rman DEFAULT CHARACTER SET utf8;   #Reports Manager
    mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password';

    mysql> create database metastore DEFAULT CHARACTER SET utf8;  #hive
    mysql> grant all on metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive_password';

    mysql> create database sentry DEFAULT CHARACTER SET utf8;
    mysql> grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password';

    mysql> create database nav DEFAULT CHARACTER SET utf8;    #Navigator Audit Server
    mysql> grant all on nav.* TO 'nav'@'%' IDENTIFIED BY 'nav_password';

    mysql> create database navms DEFAULT CHARACTER SET utf8;  #Navigator Metadata Server
    mysql> grant all on navms.* TO 'navms'@'%' IDENTIFIED BY 'navms_password';

    mysql>create database oozie DEFAULT CHARACTER SET utf8;   #Oozie Server
    mysql>grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password';

    mysql>update user set authentication_string=password('root') where user='root';
    mysql>Grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;

    mysql> create database hue DEFAULT CHARACTER SET utf8;  #hue
    mysql> grant all on hue.* TO 'hue'@'%' IDENTIFIED BY 'hue_password';

    mysql>flush privileges;
    mysql>exit

Cloudera Manager 安装

Cloudera Manager可以说是Cloudera系列产品和组件的核心,它负责统一配置管理CDH集群,基本上本篇文章都是围绕CM的安装部署来展开的,然后再通过CM去安装CDH及其其它相关组件。 
在master节点安装Cloudera Manager Server并启动。
$ yum install -y cloudera-manager-daemons cloudera-manager-server

配置文件目录

/etc/cloudera-scm-server

初始化mysql信息 
[root@cyyy-4 cmf]# whereis cmf
cmf: /usr/lib64/cmf /usr/share/cmf
得到目录/usr/share/cmf
正式初始化

/usr/share/cmf/schema/scm_prepare_database.sh mysql -hcdh1 -uroot -proot --scm-host cdh1 scmdbn scmdbu scmdbp

$ systemctl start cloudera-scm-server

日记
/var/log/cloudera-scm-server

在master和node节点安装Cloudera Manager Agent

CM Agent负责启动和杀死各个CM管理的服务进程,解压和安装CM管理的服务,自动配置,通过心跳机制监视集群服务器状态(默认每15秒Agent发送心跳给CM Server)等
修改 /etc/cloudera-scm-agent/config.ini 中的server_host为master的IP。(192.168.0.1)
$ yum install -y cloudera-manager-agent cloudera-manager-daemons
$ systemctl start cloudera-scm-agent

日记
/var/log/cloudera-scm-agent

访问
http://cdh1:7180

admin admin



hue
http://cdh1:8888

admin admin



停止服务
systemctl stop cloudera-scm-agent
systemctl stop cloudera-scm-server


安装参考:
https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_cdh5_install.html
https://zhangchenchen.github.io/2018/01/26/centos7-install-cdh5.14/

阅读更多
想对作者说点什么?

博主推荐

换一批

没有更多推荐了,返回首页