CDH,全称Cloudera's Distribution, including Apache Hadoop。是Hadoop众多分支中对应中的一种,由Cloudera维护,基于稳定版本的Apache Hadoop构建,提供了Hadoop的核心(可扩展存 储、分布式计算),最为重要的是提供基于web的用户界面。
CDH的优点:版本划分清晰,更新速度快,支持Kerberos安全认证,支持多种安装方式(如Yum、rpm等)。
CDH分为Cloudera Manager管理平台和CDH parcel(parcel包含各种组件的安装包)。这里采用CDH6.3.2。
安装前置需求:hosts,hostname,防火墙关闭,selinux关闭,ssh免密登录,自带的mariadb卸载,NTP时间同步,swap关闭,jdk安装,透明页禁用
CM6.3.1
https://archive.cloudera.com/cm6/6.3.1/
下载地址:https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4210.00MBcloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:421.12GBcloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4211.22KBcloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4210.74KBenterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm2019-10-11 08:4213.55MBoracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm2019-10-11 08:42176.42MB
allkeys.asc | 2019-10-11 08:42 | 13.71KB |
CDH6.3.2
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | 2019-11-12 13:43 | 1.94GB |
manifest.json2019-11-12 13:4333.09KB
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 | 2019-11-12 13:43 | 40B |
mysql-connector.jar
下载地址:https://cdn.mysql.com/archives/mysql-connector-java-5.1/mysql-connector-java-5.1.46.tar.gz
mysql-connector-java-5.1.46.tar.gz
wget https://cdn.mysql.com/archives/mysql-connector-java-5.1/mysql-connector-java-5.1.46.tar.gz
mkdir /usr/share/java
tar -zxvf mysql-connector-java-5.1.46.tar.gz
mv mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar /usr/share/java
#scp分发至其他节点
ps①:如果遇到ERROR JDBC Driver com.mysql.jdbc.Driver not found问题,将jar放在/opt/cloudera/cm/schema/../lib/
ps②:如果HIVE遇到Hive Metastore canary问题,可能缺少jar包,将mysql-connector-java-5.1.46.jar复制到/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib目录下
位置可通过ps aux | grep查看
通过yum安装(不推荐,国内速度慢)
sudo wget https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
sudo rpm --import https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPM-GPG-KEY-cloudera
配置cloudera yum本地源
下载以下文件
可选项
cloudera-manager-agent.rpm
cloudera-manager-daemons.rpm
cloudera-manager-server.rpm
cloudera-manager-server-db.rpm
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
直接从https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/下载对应gz包即可创建本地离线cloudera库
将下载的文件解压移动至/var/www/html/cloudera-repos下
tar xvfz cm6.3.1-redhat7.tar.gz -C /var/www/html/cloudera-repos/cm6.3.1 --strip-components=1
allkeys.asc放入/var/www/html/cloudera-repos/cm6.3.1 |
在各个节点配置yum源
vim /etc/yum.repos.d/cloudera-repo.repo
[cloudera-repo]
name=cloudera-repo
baseurl=http://node01/cloudera-repos/cm6.3.1/
enabled=1
gpgcheck=0
yum clean all
yum makecache
CM server or agent安装
不推荐
# 将安装包上传到目标服务器,在安装包目录执行安装即可
# 主节点通过 yum 安装
sudo yum install -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
sudo yum install -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
sudo yum install -y cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
# 从节点通过 yum 安装
sudo yum install -y cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
sudo yum install -y cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
yum install oracle-j2sdk1.8.x86_64 cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server#不用copy到cloudera-manager.repo到其他节点,其他节点不用安装cloudera-manager-agent,图形界面的时候,会自动copy,并安装。
my.cnf官方推荐配置
# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
创建数据库,并分配权限,mysql节点
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456';
通过官方提供了一个脚本初始化 CM 相关的数据
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh -h node03 mysql scm scm
签名
将CDH 离线包CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel放入/opt/cloudera/parcel-repo
在/opt/cloudera/parcel-repo目录生成sha1
签名文件
sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk '{ print $1 }' > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
可以通过systemctl start cloudera-scm-server和systemctl enable cloudera-scm-agent启动服务
CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha1修改为CDH-6.3.1-1.cdh6.3.1.p0.1470567-el7.parcel.sha放入/opt/cloudera/parcel-repo
可以通过tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
观看日志,当看到Started Jetty server.
时表示服务已经启动成功了,访问对应IP+7180端口即可
![](https://i-blog.csdnimg.cn/blog_migrate/f78e6832af684f34be9cd04ab0e5115b.png)
![](https://i-blog.csdnimg.cn/blog_migrate/d687cafcc562338d4195df30d88830ee.png)