CDH 6.0.1安裝

CDH安装文档

• 环境准备

1. 软件版本选择

类目	版本	下载地址
操作系统	CentOS Linux release 7.5.1804 (Core)	https://mirrors.aliyun.com/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso
数据库	mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar	http://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar
JDK	JDK 1.8	https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
Cloudera Manager	cloudera-manager-server-6.0.1	https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
	cloudera-manager-daemons-6.0.1	https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
	cloudera-manager-agent-6.0.1	https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
CDH	CDH 6.0.1	https://archive.cloudera.com/cdh6/6.0.1/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
	manifest.json	https://archive.cloudera.com/cdh6/6.0.1/parcels/manifest.json

 

1. 节点

IP	hostname	内存	磁盘	node	CM管理软件
192.168.1.81	CDH1	8G	50G	Master	Cloudera Manager Server&Agent ，MYSQL
192.168.1.82	CDH2	8G	50G	Worker	Cloudera Manager Agent
192.168.1.83	CDH3	8G	50G	Worker	Cloudera Manager Agent
192.168.1.84	CDH4	8G	50G	Worker	Cloudera Manager Agent

1. 上传软件包

1. 创建软件包上传目录

              mkdir -p /usr/local/software

              cd /usr/local/software

 

1. 上传软件包，软件包清单
   1. 1. CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
      2. cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
      3. cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
      4. cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
      5. manifest.json
      6. mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar
      7. mysql-connector-java-8.0.13.tar.gz
      8. oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

1. 配置主机名和hosts解析(所有节点)

编辑/etc/hostname，修改主机名，并使用命令hostnamectl使其立刻生效。
编辑文件/etc/hosts，增加hosts内容：

       vi /etc/hostname

       hostnamectl

 

       vi /etc/hosts

 

1. 关闭防火墙(所有节点)

systemctl stop firewalld.service

systemctl disable firewalld.service

1. 关闭SELinux

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

setenforce 0

1. 添加定时任务(时间同步，可以添加NTP服务)

echo "$((RANDOM%60)) $((RANDOM%24)) * * * /usr/sbin/ntpdate time1.aliyun.com" >> /var/spool/cron/root

1. 禁用透明大页面压缩

CDH配置需要

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

并将上面的两条命令写入开机自启动/etc/rc.local

1. 优化交换分区

echo "vm.swappiness = 10" >> /etc/sysctl.conf

sysctl -p

• 安装CM和CDH

1. 配置 JDK (所有节点)

rpm -ivh oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

 

1. 安装 CM Server 和 Agent

Master:

rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm

rpm -ivh cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm

rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm

安装agent报错缺少依赖，解决办法：yum install psmisc  cyrus-sasl-gssapi /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python[11] 

Worker

rpm -ivh cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm

rpm -ivh cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm

1. 安装mysql（Master）

mkdir mysql

mv  mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar mysql/

cd mysql

tar xvf  mysql-5.7.24-1.el7.x86_64.rpm-bundle.tar

安装mysql

rpm -ivh mysql-community-common-5.7.24-1.el7.x86_64.rpm --nodeps --force

rpm -ivh mysql-community-libs-5.7.24-1.el7.x86_64.rpm --nodeps --force

rpm -ivh mysql-community-client-5.7.24-1.el7.x86_64.rpm --nodeps --force

rpm -ivh mysql-community-server-5.7.24-1.el7.x86_64.rpm --nodeps --force

启动、停止：

service mysqld start

service mysqld stop

service mysqld status

 

修改数据库配置文件 /etc/my.conf

vi /etc/my.conf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

transaction-isolation = READ-COMMITTED

# Disabling symbolic-links is recommended to prevent assorted security risks;

# to do so, uncomment this line:

symbolic-links = 0

 

key_buffer_size = 32M

max_allowed_packet = 32M

thread_stack = 256K

thread_cache_size = 64

query_cache_limit = 8M

query_cache_size = 64M

query_cache_type = 1

 

max_connections = 550

#expire_logs_days = 10

#max_binlog_size = 100M

 

#log_bin should be on a disk with enough free space.

#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your

#system and chown the specified folder to the mysql user.

log_bin=/var/lib/mysql/mysql_binary_log

 

#In later versions of MySQL, if you enable the binary log and do not set

#a server_id, MySQL will not start. The server_id must be unique within

#the replicating group.

server_id=1

 

binlog_format = mixed

 

read_buffer_size = 2M

read_rnd_buffer_size = 16M

sort_buffer_size = 8M

join_buffer_size = 8M

 

# InnoDB settings

innodb_file_per_table = 1

innodb_flush_log_at_trx_commit  = 2

innodb_log_buffer_size = 64M

innodb_buffer_pool_size = 4G

innodb_thread_concurrency = 8

innodb_flush_method = O_DIRECT

innodb_log_file_size = 512M

 

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

 

sql_mode=STRICT_ALL_TABLES

添加到自启动项

sudo systemctl enable mysqld

重启MYSQL

       sudo systemctl start mysqld

修改root密码

       cat /var/log/mysqld.log | more

连接mysql

mysql -uroot -p

修改密码复杂度：

set global validate_password_policy=0;

set global validate_password_policy=0;

set global validate_password_mixed_case_count=0;

set global validate_password_number_count=3;

set global validate_password_special_char_count=0;

set global validate_password_length=3;

设置密码      

set password='123456';

grant all privileges on *.* to 'root'@'%' identified by '123456';

 

 

 

 

 

 

 

 

 

 

 

 

为 Cloudera 各软件创建数据库

使用root创建以下数据库和账号。

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY  'hems123';

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'hems123';

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'hems123';

安装 MySQL JDBC 驱动(所有节点)

       mkdir -p /usr/share/java/

tar zxvf mysql-connector-java-8.0.13.tar.gz

cd mysql-connector-java-8.0.13/

cp mysql-connector-java-8.0.13.jar /usr/share/java/mysql-connector-java.jar

设置 Cloudera Manager 数据库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

 

注：

该命令是当数据库和Cloudera Manager Server装在同一个节点上时，在该节点上执行使用的。其他部署情况以及脚本的详细使用方法请参考：https://www.cloudera.com/documentation/enterprise/6/6.0/topics/prepare_cm_database.html

配置CDH的软件包 parcels（Master）

       在manifest.json文件中，找到对应版本的秘钥，复制到.sha文件中。

echo "2e650f1f1ea020a3efc98a231b85c2df1a50b030" > CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha

cp  CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel /opt/cloudera/parcel-repo

cp CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha /opt/cloudera/parcel-repo

修改属主属组。

chown cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/*

启动 Cloudera Manager Server

       systemctl start cloudera-scm-server

注：

如果启动中有什么问题，可以查看日志。

      tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

 

---

 [11]此处暂时没有做离线安装，可以下载rpm包安装