总结:
- 配置cm-server的yum本地安装路径,使用yum安装好cm-server即可,cm-agent无需手动到节点去安装,cm会自动在节点上安装。
- parcels文件不用安装yum本地源,后续直接拷贝到cm server 的/opt/cloudera/parcel-repo/下即可。
- jdk手动安装,目录必须是:/usr/java/jdk-version,否则报错
- yum需要安装前期的依赖,否则后面agent无法安装,出现莫名其妙的问题,安装依赖后就成功了!
服务器设置
说明:服务器centos7
java version “1.8.0_251”
1.安装依赖
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb -y
关闭防火墙
sudo systemctl disable firewalld
sudo systemctl stop firewalld
2.配置系统环境
以下需要在每个节点上执行
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'vm.swappiness=10'>> /etc/sysctl.conf # 这个也要重启
# 关闭Security-Enhanced Linux (SELinux)
# 查询是否关闭
getenforce
# 修改/etc/selinux/config
SELINUX=disabled
# 重启
3.开启NTP服务
yum install ntp
编辑/etc/ntp.conf
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
开启服务
sudo systemctl start ntpd
sudo systemctl enable ntpd
# 其他机器可同步dc5同步时间
ntpdate -u <ntp_server>
# 写入系统时间
hwclock --systohc
4.CDH端口情况一览
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_ports.html
5.安装jdk
jdk必须是64bit,注意jdk目录必须是/usr/java/jdkxxxx ,否则安装server将失败
6.安装数据库
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
systemctl start mysqld
systemctl enable mysqld
# 设置密码
/usr/bin/mysql_secure_installation
7.安装数据库驱动
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar
# 拷贝到目录,如果目录不存在,则新建,并且改名为mysql-connector-java.jar
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
8.执行数据库脚本
执行脚本:
# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>
/opt/cloudera/cm/schema/scm_prepare_database.sh -uroot -proot mysql scm scm scm123
# 出现All done, your SCM database is configured correctly!标识成功
创建其他数据库(以后安装服务的时候还是得创建)
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
-- 注意密码最低要求4位以上
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123';
开始安装
离线安装
CDH提供2种方式安装:parcel repositories and package repositories.
安装cloudera manager 要使用yum来安装,6以前可以使用tgz安装,6以后没有tgz包安装了
本地yum安装
本地yum安装cloudera manager
1.安装httpd服务
sudo yum install httpd
sudo systemctl start httpd
2.下载文件
sudo mkdir -p /var/www/html/cloudera-repos/cm6
# 注意cm目前6版本只有6.3.1,cm服务文件可以是6.3.2
# 注意目录是除host外,其他保持一致
sudo mkdir -p /var/www/html/cloudera-repos/
将6个文件全部下载到/var/www/html/cloudera-repos/ 这个目录即可(不用创建和下载一样的路径)
下载地址:https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
同时下载https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/ 路径下的cloudera-manager.repo和RPM-GPG-KEY-cloudera
chmod -R ugo+rX /var/www/html/cloudera-repos/
# 在当前目录下执行
createrepo . # 生成yum元数据进行yum安装即可
3.配置本地源
vim cloudera-manager.repo
[cloudera-manager]
name=Cloudera Manager 6.3.2
baseurl=http://dc5/cloudera-repos/yum # 对应到上面rpm本地目录
gpgcheck=0
enabled=1
并将yum配置分发到每个agent节点。
执行yum clean all & yum makecache
yum repolist 查看是否生效
4.开始安装cloddera manager package
yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
5.在server之外节点安装agent
yum install cloudera-manager-agent cloudera-manager-daemons
6.启动server和agent
systemctl start cloudera-scm-server
systemctl stop cloudera-scm-server
systemctl start cloudera-scm-agent
systemctl stop cloudera-scm-agent
- 查看日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log
当出现如下执行ddl脚本,表示启动server成功
2020-05-08 14:20:34,367 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,368 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,376 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,377 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 0
2020-05-08 14:20:40,148 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00001_cmf_schema.mysql.ddl
2020-05-08 14:20:40,160 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 1
2020-05-08 14:20:40,458 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00002_cmf_schema.mysql.ddl
2020-05-08 14:20:40,470 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 2
2020-05-08 14:20:40,628 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00003_cmf_schema.mysql.ddl
2020-05-08 14:20:40,639 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 3
2020-05-08 14:20:40,805 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00004_cmf_schema.mysql.ddl
8.访问admin界面:
http://dc5:7180 输入密码:admin/admin
本地离线parcel
这个步骤可以直接省略,把https://archive.cloudera.com/cdh6/6.3.2/parcels/下载的4个文档放到server中的/opt/cloudera/parcel-repo的目录下即可,如果无效,然后再使用httpd来添加到自定义存储库
使用这种方式前提是已经安装好cloudera manager
1.配置本地parcel repositories
sudo yum install httpd
修/etc/httpd/conf/httpd.conf 在<IfModule mime_module>
增加:
AddType application/x-gzip .gz .tgz .parcel
启动httpd服务
sudo systemctl start httpd
2.下载parcel包
下载地址: https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_63_download.html#cdh_633-download
目前新版本6.3.3需要用户名密码下载,这里用6.3.2版本
总共下载4个文件,放入httpd服务器的目录,这样就能用使用本地httpd提供服务来分发parcel包
# cdh文件
mkdir -p /var/www/html/cloudera-repos/cdh6/6.3.2/parcels/
cd /var/www/html/cloudera-repos/cdh6/6.3.2/parcels/
wegt https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
# 这个文件下载后要把后面的1给去掉,否则会从网上下载,而不是使用本地
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 -P CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json
# gplextras文件
mkdir -p /var/www/html/cloudera-repos/gplextras6/6.3.2/parcels/
cd /var/www/html/cloudera-repos/gplextras6/6.3.2/parcels/
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel.sha1
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel.sha256
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/manifest.json
递归下载文件夹,忽略父文件夹命名参考:
wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/version/parcels/ -P /var/www/html/cloudera-repos
修改权限
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6
其他组件的parcel包根据需要下载,这里下载spark2
sudo mkdir -p /var/www/html/cloudera-repos
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/spark2/parcels/2.3.0.cloudera3/ -P /var/www/html/cloudera-repos
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/spark2
访问地址看看下载的文件是否都有:http://<Web_server>/cloudera-repos/
4.配置Cloudera Manager以使用本地parcels
使用这种方式也得先把cloudera manager安装好
在cdh管理页面打开导航栏中:
Administration > Settings
Category > Parcels
输入本地的地址:http://dc5/cloudera-parcels/cdh6/6.3.2/
输入变更原因,点击保存即可。
配置机器
登录cmf管理页面进行安装
注意,这里agent,可以直接游cm-server来安装。
安装spark2
注意:默认6.3.2 的spark已经是2.4.0版本了,所以已经不用再安装了
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html
https://docs.cloudera.com/documentation/spark2/latest/topics/spark2.html
下载文件到/opt/cloudera/csd
wget http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.4.0.cloudera2.jar
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha1
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/manifest.json
mv SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha1 SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha
重启server服务
systemctl restart cloudera-scm-server
主机-parcel-分配-激活
添加服务即可。
整合Flink
2020年5月7日 15:53:58截至目前官网支持flink1.9.1版本,但是下载要登陆,而是支持cdh7.0.3
文档见:https://docs.cloudera.com/csa/1.1.0/support-matrix/topics/csa-system-req.html
最新版本的需要用户名密码。
这里使用FLINK-1.9.0-csa1.0.0.0-cdh6.3.0版本,具体步骤:
参考:https://cloud.tencent.com/developer/article/1536518
下载地址:https://archive.cloudera.com/csa/
mkdir -p /var/www/html/flink1.9
cd /var/www/html/flink1.9
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/manifest.json
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0-el7.parcel
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0-el7.parcel.sha
# 进入csd,下载jar包
cd /opt/cloudera/csd
wget https://archive.cloudera.com/csa/1.0.0.0/csd/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0.jar
重启服务:systemctl restart cloudera-scm-server
主机-parcel -配置-远程parcel存储库url添加一个:http://dc5/flink1.9/ 保存