Flink v1.12 on YARN

版本号说明

Flink v1.12.x

hadoop 2.10.x

第1节 YARN环境准备

见《hadoop分布式集群搭建》篇

第2节 安装flink

（1）deploy用户登陆hadoop219

（2）上传flink-1.12.1-bin-scala_2.11.tgz

从官网下载：flink-1.12.1-bin-scala_2.11.tgz

上传到/opt/software

[deploy@hadoop219 software]$ cd /opt/software/

[deploy@hadoop219 software]$ rz

 

 

解压flink-1.12.1-bin-scala_2.11.tgz到/opt/module

[deploy@hadoop219 software]$ tar -zxvf flink-1.12.1-bin-scala_2.11.tgz -C /opt/module/

（3）上传flink-shaded-hadoop-2-uber-2.8.3-10.0.jar

注意：flink-hadoop lib包的版本号可以与yarn对应的hadoop版本不一致，hadoop版本号为2.10.x

从官网下载额外组件：Pre-bundled Hadoop 2.8.3

上传到/opt/module/flink-1.12.1/lib

[deploy@hadoop219 lib]$ cd /opt/module/flink-1.12.1/lib
[deploy@hadoop219 lib]$ rz

 

第3节 配置HADOOP_CLASSPATH环境变量

[deploy@hadoop219 flink-1.12.1]$ su
[deploy@hadoop219 flink-1.12.1]$ vi /etc/profile
#在末尾增加
export HADOOP_CLASSPATH=`hadoop classpath`
[root@hadoop219 flink-1.12.1]# source /etc/profile
[root@hadoop219 flink-1.12.1]# exit
[deploy@hadoop219 flink-1.12.1]$ source /etc/profile
[deploy@hadoop219 flink-1.12.1]$ echo $HADOOP_CLASSPATH
/opt/module/hadoop-2.10.1/etc/hadoop:/opt/module/hadoop-2.10.1/share/hadoop/common/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/common/*:/opt/module/hadoop-2.10.1/share/hadoop/hdfs:/opt/module/hadoop-2.10.1/share/hadoop/hdfs/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/hdfs/*:/opt/module/hadoop-2.10.1/share/hadoop/yarn:/opt/module/hadoop-2.10.1/share/hadoop/yarn/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/yarn/*:/opt/module/hadoop-2.10.1/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/mapreduce/*:/opt/module/hadoop-2.10.1/contrib/capacity-scheduler/*.jar

 

第4节 在YARN上启动Flink Session Cluster

在yarn nodemanager节点执行都可以

在hadoop219上启动Flink Session Cluster

[deploy@hadoop219 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop219 flink-1.12.1]$ bin/yarn-session.sh --detached

yarn管理页面查看flink集群

 

查看flink管理页面

 

提交测试job

[deploy@hadoop219 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop219 flink-1.12.1]$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
#flink管理页面查看job执行状态

 停止测试job

[deploy@hadoop219 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop219 flink-1.12.1]$ echo "stop" | ./bin/yarn-session.sh -id application_1614213970091_0001
#flink session cluster结束运行

第5节 java flink工程maven打包

maven配置打包，pom.xml配置内容如下

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <!-- 可选，可以不配置mainClass，不配置在部署flink时命令行里边指定mainClass -->
                        <manifest>
                            <mainClass>com.wanshun.wycapm.httpapiapm.HttpApiApmApplication</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

第6节 在YARN上启动Per Job Cluster

在yarn nodemanager节点执行都可以

在hadoop218上启动Flink Per Job Cluster

#配置HADOOP_CLASSPATH环境变量，详见第3节
[deploy@hadoop218 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop218 flink-1.12.1]$ mkdir app
[deploy@hadoop218 flink-1.12.1]$ cd app
[deploy@hadoop218 flink-1.12.1]$ rz
#上传打包的flink应用httpApiApm-1.0-SNAPSHOT-jar-with-dependencies.jar
[deploy@hadoop218 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop218 flink-1.12.1]$ ./bin/flink run -t yarn-per-job --detached ./app/httpApiApm-1.0-SNAPSHOT-jar-with-dependencies.jar

查看Cluster信息

[deploy@hadoop218 flink-1.12.1]$ ./bin/flink list -t yarn-per-job -Dyarn.application.id=application_1614213970091_0004

停止Flink Per Job Cluster 

./bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_XXXX_YY <jobId>

第七节 Flink demo工程

flink-apm: flink在yarn上运行的demo