cdh6.3安装以及整合spark2、flink1.9

总结:

  1. 配置cm-server的yum本地安装路径,使用yum安装好cm-server即可,cm-agent无需手动到节点去安装,cm会自动在节点上安装。
  2. parcels文件不用安装yum本地源,后续直接拷贝到cm server 的/opt/cloudera/parcel-repo/下即可。
  3. jdk手动安装,目录必须是:/usr/java/jdk-version,否则报错
  4. yum需要安装前期的依赖,否则后面agent无法安装,出现莫名其妙的问题,安装依赖后就成功了!

服务器设置

说明:服务器centos7
java version “1.8.0_251”

1.安装依赖

yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb -y

关闭防火墙

sudo systemctl disable firewalld
sudo systemctl stop firewalld

2.配置系统环境

以下需要在每个节点上执行

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'vm.swappiness=10'>> /etc/sysctl.conf   #  这个也要重启

# 关闭Security-Enhanced Linux (SELinux)
# 查询是否关闭
getenforce 
# 修改/etc/selinux/config
SELINUX=disabled
# 重启

3.开启NTP服务

yum install ntp

编辑/etc/ntp.conf

server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org

开启服务

sudo systemctl start ntpd
sudo systemctl enable ntpd
# 其他机器可同步dc5同步时间
ntpdate -u <ntp_server>
# 写入系统时间
hwclock --systohc

4.CDH端口情况一览
https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_ports.html

5.安装jdk

jdk必须是64bit,注意jdk目录必须是/usr/java/jdkxxxx ,否则安装server将失败

6.安装数据库

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
systemctl start mysqld
systemctl enable mysqld

# 设置密码
/usr/bin/mysql_secure_installation

7.安装数据库驱动

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar
# 拷贝到目录,如果目录不存在,则新建,并且改名为mysql-connector-java.jar
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

8.执行数据库脚本

执行脚本:

# sudo /opt/cloudera/cm/schema/scm_prepare_database.sh [options] <databaseType> <databaseName> <databaseUser> <password>

/opt/cloudera/cm/schema/scm_prepare_database.sh -uroot -proot  mysql scm scm scm123

# 出现All done, your SCM database is configured correctly!标识成功

创建其他数据库(以后安装服务的时候还是得创建)

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

-- 注意密码最低要求4位以上
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm123';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie123';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue123';

开始安装

离线安装
CDH提供2种方式安装:parcel repositories and package repositories.

安装cloudera manager 要使用yum来安装,6以前可以使用tgz安装,6以后没有tgz包安装了

本地yum安装

本地yum安装cloudera manager

1.安装httpd服务

sudo yum install httpd
sudo systemctl start httpd

2.下载文件

sudo mkdir -p /var/www/html/cloudera-repos/cm6
# 注意cm目前6版本只有6.3.1,cm服务文件可以是6.3.2
# 注意目录是除host外,其他保持一致
sudo mkdir -p /var/www/html/cloudera-repos/

将6个文件全部下载到/var/www/html/cloudera-repos/ 这个目录即可(不用创建和下载一样的路径)
下载地址:https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/
同时下载https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/ 路径下的cloudera-manager.repo和RPM-GPG-KEY-cloudera	

chmod -R ugo+rX /var/www/html/cloudera-repos/
# 在当前目录下执行
createrepo . # 生成yum元数据进行yum安装即可

3.配置本地源
vim cloudera-manager.repo

[cloudera-manager]
name=Cloudera Manager 6.3.2
baseurl=http://dc5/cloudera-repos/yum   # 对应到上面rpm本地目录
gpgcheck=0
enabled=1

并将yum配置分发到每个agent节点。
执行yum clean all & yum makecache
yum repolist 查看是否生效

4.开始安装cloddera manager package

yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

5.在server之外节点安装agent

yum install cloudera-manager-agent cloudera-manager-daemons

6.启动server和agent

systemctl start cloudera-scm-server
systemctl stop  cloudera-scm-server

systemctl start cloudera-scm-agent
systemctl stop cloudera-scm-agent
  1. 查看日志
 tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
 tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

当出现如下执行ddl脚本,表示启动server成功

2020-05-08 14:20:34,367 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,368 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,376 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2020-05-08 14:20:34,377 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 0
2020-05-08 14:20:40,148 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00001_cmf_schema.mysql.ddl
2020-05-08 14:20:40,160 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 1
2020-05-08 14:20:40,458 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00002_cmf_schema.mysql.ddl
2020-05-08 14:20:40,470 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 2
2020-05-08 14:20:40,628 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00003_cmf_schema.mysql.ddl
2020-05-08 14:20:40,639 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: Updated Schema Version to 3
2020-05-08 14:20:40,805 INFO main:com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlStringRunner: Processed: /opt/cloudera/cm/schema/mysql/00004_cmf_schema.mysql.ddl

8.访问admin界面:

http://dc5:7180 输入密码:admin/admin

本地离线parcel

这个步骤可以直接省略,把https://archive.cloudera.com/cdh6/6.3.2/parcels/下载的4个文档放到server中的/opt/cloudera/parcel-repo的目录下即可,如果无效,然后再使用httpd来添加到自定义存储库

使用这种方式前提是已经安装好cloudera manager

1.配置本地parcel repositories

sudo yum install httpd

修/etc/httpd/conf/httpd.conf 在<IfModule mime_module>增加:

AddType application/x-gzip .gz .tgz .parcel

启动httpd服务

sudo systemctl start httpd

2.下载parcel包

下载地址: https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_63_download.html#cdh_633-download

目前新版本6.3.3需要用户名密码下载,这里用6.3.2版本

总共下载4个文件,放入httpd服务器的目录,这样就能用使用本地httpd提供服务来分发parcel包

# cdh文件
mkdir -p /var/www/html/cloudera-repos/cdh6/6.3.2/parcels/
cd /var/www/html/cloudera-repos/cdh6/6.3.2/parcels/

wegt https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel

# 这个文件下载后要把后面的1给去掉,否则会从网上下载,而不是使用本地
wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 -P CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

wget https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256

wget https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json

# gplextras文件
mkdir -p /var/www/html/cloudera-repos/gplextras6/6.3.2/parcels/
cd /var/www/html/cloudera-repos/gplextras6/6.3.2/parcels/

wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel.sha1
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/GPLEXTRAS-6.3.2-1.gplextras6.3.2.p0.1605554-el7.parcel.sha256
wget https://archive.cloudera.com/gplextras6/6.3.2/parcels/manifest.json

递归下载文件夹,忽略父文件夹命名参考:
wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/gplextras6/version/parcels/ -P /var/www/html/cloudera-repos

修改权限

sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

其他组件的parcel包根据需要下载,这里下载spark2

sudo mkdir -p /var/www/html/cloudera-repos
sudo wget --recursive --no-parent --no-host-directories https://archive.cloudera.com/spark2/parcels/2.3.0.cloudera3/ -P /var/www/html/cloudera-repos
sudo chmod -R ugo+rX /var/www/html/cloudera-repos/spark2

访问地址看看下载的文件是否都有:http://<Web_server>/cloudera-repos/

4.配置Cloudera Manager以使用本地parcels

使用这种方式也得先把cloudera manager安装好

在cdh管理页面打开导航栏中:
Administration > Settings
Category > Parcels

输入本地的地址:http://dc5/cloudera-parcels/cdh6/6.3.2/

输入变更原因,点击保存即可。

配置机器

登录cmf管理页面进行安装

注意,这里agent,可以直接游cm-server来安装。

安装spark2

注意:默认6.3.2 的spark已经是2.4.0版本了,所以已经不用再安装了

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html

https://docs.cloudera.com/documentation/spark2/latest/topics/spark2.html

下载文件到/opt/cloudera/csd

wget http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.4.0.cloudera2.jar

wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha1
wget http://archive.cloudera.com/spark2/parcels/2.4.0.cloudera2/manifest.json

mv SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha1 SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012-el7.parcel.sha

重启server服务

systemctl restart cloudera-scm-server

主机-parcel-分配-激活
添加服务即可。

整合Flink

2020年5月7日 15:53:58截至目前官网支持flink1.9.1版本,但是下载要登陆,而是支持cdh7.0.3
文档见:https://docs.cloudera.com/csa/1.1.0/support-matrix/topics/csa-system-req.html

最新版本的需要用户名密码。

这里使用FLINK-1.9.0-csa1.0.0.0-cdh6.3.0版本,具体步骤:

参考:https://cloud.tencent.com/developer/article/1536518

下载地址:https://archive.cloudera.com/csa/

mkdir -p /var/www/html/flink1.9
cd /var/www/html/flink1.9
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/manifest.json
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0-el7.parcel
wget https://archive.cloudera.com/csa/1.0.0.0/parcels/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0-el7.parcel.sha

# 进入csd,下载jar包
cd /opt/cloudera/csd
wget https://archive.cloudera.com/csa/1.0.0.0/csd/FLINK-1.9.0-csa1.0.0.0-cdh6.3.0.jar

重启服务:systemctl restart cloudera-scm-server
主机-parcel -配置-远程parcel存储库url添加一个:http://dc5/flink1.9/ 保存
在这里插入图片描述

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值