版本号说明
Flink v1.12.x
hadoop 2.10.x
第1节 YARN环境准备
见《hadoop分布式集群搭建》篇
第2节 安装flink
(1)deploy用户登陆hadoop219
(2)上传flink-1.12.1-bin-scala_2.11.tgz
从官网下载:flink-1.12.1-bin-scala_2.11.tgz
上传到/opt/software
[deploy@hadoop219 software]$ cd /opt/software/
[deploy@hadoop219 software]$ rz
解压flink-1.12.1-bin-scala_2.11.tgz到/opt/module
[deploy@hadoop219 software]$ tar -zxvf flink-1.12.1-bin-scala_2.11.tgz -C /opt/module/
(3)上传flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
注意:flink-hadoop lib包的版本号可以与yarn对应的hadoop版本不一致,hadoop版本号为2.10.x
从官网下载额外组件:Pre-bundled Hadoop 2.8.3
上传到/opt/module/flink-1.12.1/lib
[deploy@hadoop219 lib]$ cd /opt/module/flink-1.12.1/lib
[deploy@hadoop219 lib]$ rz
第3节 配置HADOOP_CLASSPATH环境变量
[deploy@hadoop219 flink-1.12.1]$ su
[deploy@hadoop219 flink-1.12.1]$ vi /etc/profile
#在末尾增加
export HADOOP_CLASSPATH=`hadoop classpath`
[root@hadoop219 flink-1.12.1]# source /etc/profile
[root@hadoop219 flink-1.12.1]# exit
[deploy@hadoop219 flink-1.12.1]$ source /etc/profile
[deploy@hadoop219 flink-1.12.1]$ echo $HADOOP_CLASSPATH
/opt/module/hadoop-2.10.1/etc/hadoop:/opt/module/hadoop-2.10.1/share/hadoop/common/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/common/*:/opt/module/hadoop-2.10.1/share/hadoop/hdfs:/opt/module/hadoop-2.10.1/share/hadoop/hdfs/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/hdfs/*:/opt/module/hadoop-2.10.1/share/hadoop/yarn:/opt/module/hadoop-2.10.1/share/hadoop/yarn/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/yarn/*:/opt/module/hadoop-2.10.1/share/hadoop/mapreduce/lib/*:/opt/module/hadoop-2.10.1/share/hadoop/mapreduce/*:/opt/module/hadoop-2.10.1/contrib/capacity-scheduler/*.jar
第4节 在YARN上启动Flink Session Cluster
在yarn nodemanager节点执行都可以
在hadoop219上启动Flink Session Cluster
[deploy@hadoop219 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop219 flink-1.12.1]$ bin/yarn-session.sh --detached
yarn管理页面查看flink集群
查看flink管理页面
提交测试job
[deploy@hadoop219 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop219 flink-1.12.1]$ ./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
#flink管理页面查看job执行状态
停止测试job
[deploy@hadoop219 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop219 flink-1.12.1]$ echo "stop" | ./bin/yarn-session.sh -id application_1614213970091_0001
#flink session cluster结束运行
第5节 java flink工程maven打包
maven配置打包,pom.xml配置内容如下
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<!-- 可选,可以不配置mainClass,不配置在部署flink时命令行里边指定mainClass -->
<manifest>
<mainClass>com.wanshun.wycapm.httpapiapm.HttpApiApmApplication</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
第6节 在YARN上启动Per Job Cluster
在yarn nodemanager节点执行都可以
在hadoop218上启动Flink Per Job Cluster
#配置HADOOP_CLASSPATH环境变量,详见第3节
[deploy@hadoop218 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop218 flink-1.12.1]$ mkdir app
[deploy@hadoop218 flink-1.12.1]$ cd app
[deploy@hadoop218 flink-1.12.1]$ rz
#上传打包的flink应用httpApiApm-1.0-SNAPSHOT-jar-with-dependencies.jar
[deploy@hadoop218 flink-1.12.1]$ cd /opt/module/flink-1.12.1/
[deploy@hadoop218 flink-1.12.1]$ ./bin/flink run -t yarn-per-job --detached ./app/httpApiApm-1.0-SNAPSHOT-jar-with-dependencies.jar
查看Cluster信息
[deploy@hadoop218 flink-1.12.1]$ ./bin/flink list -t yarn-per-job -Dyarn.application.id=application_1614213970091_0004
停止Flink Per Job Cluster
./bin/flink cancel -t yarn-per-job -Dyarn.application.id=application_XXXX_YY <jobId>