CDH6.3.2集成flink1.13

目录

一、编译条件

二、制作parcel

三、进行编译

 四、cdh6.3.2 集成 flink

 五、解决4个报错

---

一、编译条件

jdk1.8

maven3.8.1

parcel制作工具

(注：如果需要编译好的包(34条消息) CDH6.3.2集成flink1.13parcel包-其它文档类资源-CSDN文库https://download.csdn.net/download/m0_49027804/86818597)

二、制作parcel

下载制作工具

git clone https://github.com/pkeropen/flink-parcel.git

如果下载报下面这个错误：

fatal: unable to access 'https://github.com/pkeropen/flink-parcel.git/': Encountered end of file
可以执行下面这些语句解决

git config --global http.postBuffer 50M
git config --global http.proxy
git config --global --unset http.proxy

进入flink-parcel,修改配置文件

cd ./flink-parcel
vim flink-parcel.properties
进行相应修改，内容如下：
#FLINK 下载地址
FLINK_URL=https://archive.apache.org/dist/flink/flink-1.13.2/flink-1.13.2-bin-scala_2.11.tgz

#flink版本号
FLINK_VERSION=1.13.2

#扩展版本号
EXTENS_VERSION=BIN-SCALA_2.11

#操作系统版本，以centos为例
OS_VERSION=7

#CDH 小版本
CDH_MIN_FULL=5.2
CDH_MAX_FULL=6.3.3

#CDH大版本
CDH_MIN=5
CDH_MAX=6

 FLINK_URL=https://archive.apache.org/dist/flink/flink-1.13.2/flink-1.13.2-bin-scala_2.11.tgz这个文件也可以先下载好，因为通过脚本下载的话网速比较慢，建议通过其他平台下载，下载好后直接放到flink-parcel目录下

三、进行编译

./build.sh  parcel
下载并打包完成后会在当前目录生成FLINK-1.13.2-BIN-SCALA_2.11_build文件
构建flink-yarn csd包

./build.sh csd_on_yarn

执行完成后会生成FLINK_ON_YARN-1.13.2.jar
 

 

 四、cdh6.3.2 集成 flink

配置html

mkdir -p  /var/www/html/cdh6_flink/

将FLINK-1.13.2-BIN-SCALA_2.11_build 所有的文件cp 到/var/www/html/cdh6_flink/

将FLINK-1.13.2-BIN-SCALA_2.11_build 所有的文件cp 到/opt/cloudera/parcel-repo

cp FLINK_ON_YARN-1.13.2.jar /opt/cloudera/csd/

配置parcel

 

 激活分配

 添加组件：如果没有flink，需要关闭集群，重启systemctl restart cloudera-scm-server

 

 五、解决4个报错

/opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh: line 17: rotateLogFilesWithPrefix: command not found

vim /opt/cloudera/parcels/FLINK/lib/flink/bin/config.sh

:391  加入

rotateLogFilesWithPrefix() {
    dir=$1
    prefix=$2
    while read -r log ; do
        rotateLogFile "$log"
    # find distinct set of log file names, ignoring the rotation number (trailing dot and digit)
    done < <(find "$dir" ! -type d -path "${prefix}*" | sed s/\.[0-9][0-9]*$// | sort | uniq)
}
# 旋转日志文件
rotateLogFile() {
    log=$1;
    num=$MAX_LOG_FILE_NUMBER
    if [ -f "$log" -a "$num" -gt 0 ]; then
        while [ $num -gt 1 ]; do
            prev=`expr $num - 1`
            [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num"
            num=$prev
        done
        mv "$log" "$log.$num";
    fi
}

重启后依旧报错,但没有具体的错误输出,但是通过日志,可以看到提示HBASE环境变量未设置

Flink-yarn -> 配置 -> 高级 -> Flink-yarn 服务环境高级配置代码段（安全阀）Flink-yarn（服务范围）加入以下内容即可：

HADOOP_USER_NAME=flink
HADOOP_CONF_DIR=/etc/hadoop/conf
HADOOP_HOME=/opt/cloudera/parcels/CDH
HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH/jars/*
HBASE_CONF_DIR=/etc/hbase/conf

 运行一个WordCount测试

./flink  run -t yarn-per-job /opt/cloudera/parcels/FLINK/lib/flink/examples/batch/WordCount.jar

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your 
classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
	at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Dep
loyment" section of the official Apache Flink documentation.	at org.apache.flink.yarn.executors.YarnJobClusterExecutorFactory.getExecutor(YarnJobClusterExecutorFactory.java:50)
	at org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1052)
	at org.apache.flink.client.program.ContextEnvironment.executeAsync(ContextEnvironment.java:129)
	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:70)
	at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:942)
	at org.apache.flink.api.java.DataSet.collect(DataSet.java:417)
	at org.apache.flink.api.java.DataSet.print(DataSet.java:1748)
	at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:96)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)

#在环境变量中加入（就是直接输入）

 export HADOOP_CLASSPATH=`hadoop classpath`

解决root无权限的问题

groupadd supergroup
usermod -a -G supergroup root
sudo -u hdfs hdfs dfsadmin -refreshUserToGroupsMappings