Cloudera CDH 6.3.2 Centos7离线安装

CDH，全称Cloudera's Distribution, including Apache Hadoop。是Hadoop众多分支中对应中的一种，由Cloudera维护，基于稳定版本的Apache Hadoop构建，提供了Hadoop的核心(可扩展存 储、分布式计算)，最为重要的是提供基于web的用户界面。
CDH的优点：版本划分清晰，更新速度快，支持Kerberos安全认证，支持多种安装方式(如Yum、rpm等)。
CDH分为Cloudera Manager管理平台和CDH parcel(parcel包含各种组件的安装包)。这里采用CDH6.3.2。
安装前置需求:hosts,hostname,防火墙关闭,selinux关闭,ssh免密登录,自带的mariadb卸载,NTP时间同步,swap关闭,jdk安装,透明页禁用

CM6.3.1

https://archive.cloudera.com/cm6/6.3.1/
下载地址：https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4210.00MBcloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:421.12GBcloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4211.22KBcloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4210.74KBenterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4213.55MBoracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm2019-10-11 08:42176.42MB

allkeys.asc

2019-10-11 08:42

13.71KB

CDH6.3.2

下载地址: https://archive.cloudera.com/cdh6/6.3.2/parcels/

CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel

2019-11-12 13:43

1.94GB

manifest.json2019-11-12 13:4333.09KB

CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1

2019-11-12 13:43

40B

mysql-connector.jar

下载地址：https://cdn.mysql.com/archives/mysql-connector-java-5.1/mysql-connector-java-5.1.46.tar.gz
mysql-connector-java-5.1.46.tar.gz

wget https://cdn.mysql.com/archives/mysql-connector-java-5.1/mysql-connector-java-5.1.46.tar.gz
mkdir /usr/share/java
tar -zxvf mysql-connector-java-5.1.46.tar.gz
mv mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar /usr/share/java
#scp分发至其他节点

ps①:如果遇到ERROR JDBC Driver com.mysql.jdbc.Driver not found问题，将jar放在/opt/cloudera/cm/schema/../lib/

ps②:如果HIVE遇到Hive Metastore canary问题，可能缺少jar包，将mysql-connector-java-5.1.46.jar复制到/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib目录下

位置可通过ps aux | grep查看

通过yum安装(不推荐，国内速度慢)

sudo wget https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
sudo rpm --import https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera

配置cloudera yum本地源

node01节点yum install httpd createrepo
下载以下文件

可选项

allkeys.asc
cloudera-manager-agent.rpm
cloudera-manager-daemons.rpm
cloudera-manager-server.rpm
cloudera-manager-server-db.rpm
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel

直接从https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/下载对应gz包即可创建本地离线cloudera库
将下载的文件解压移动至/var/www/html/cloudera-repos下

tar xvfz cm6.3.1-redhat7.tar.gz -C /var/www/html/cloudera-repos/cm6.3.1 --strip-components=1

allkeys.asc放入/var/www/html/cloudera-repos/cm6.3.1

在各个节点配置yum源

vim /etc/yum.repos.d/cloudera-repo.repo

[cloudera-repo]
name=cloudera-repo
baseurl=http://node01/cloudera-repos/cm6.3.1/
enabled=1
gpgcheck=0

yum clean all
yum makecache

CM server or agent安装

不推荐

# 将安装包上传到目标服务器，在安装包目录执行安装即可
# 主节点通过 yum 安装
sudo yum install -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
sudo yum install -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
sudo yum install -y cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
# 从节点通过 yum 安装
sudo yum install -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
sudo yum install -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

yum install oracle-j2sdk1.8.x86_64 cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server#不用copy到cloudera-manager.repo到其他节点，其他节点不用安装cloudera-manager-agent，图形界面的时候，会自动copy，并安装。

my.cnf官方推荐配置

# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

创建数据库，并分配权限，mysql节点

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';

通过官方提供了一个脚本初始化 CM 相关的数据

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh -h node03 mysql scm scm

签名

将CDH 离线包CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel放入/opt/cloudera/parcel-repo
在/opt/cloudera/parcel-repo目录生成sha1签名文件

sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk '{ print $1 }' > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

可以通过systemctl start cloudera-scm-server和systemctl enable cloudera-scm-agent启动服务

CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1修改为CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha放入/opt/cloudera/parcel-repo

可以通过tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log观看日志，当看到Started Jetty server.时表示服务已经启动成功了,访问对应IP+7180端口即可