Flink部署_flink workers-CSDN博客

本文链接：https://blog.csdn.net/qq_61741696/article/details/127232207

启动

简单启动

下载所需内容

[doudou@hadoop102 flink]$ ll
total 510348
-rw-r--r-- 1 doudou doudou 149499945 10月  9 04:01 elasticsearch-6.8.5.tar.gz
-rw-r--r-- 1 doudou doudou 305107189 10月  9 04:01 flink-1.13.0-bin-scala_2.12.tgz
-rw-r--r-- 1 doudou doudou  61914514 10月  9 04:01 kafka_2.11-2.1.0.tgz
-rw-r--r-- 1 doudou doudou   3597701 10月  9 04:00 mysql-connector-java-5.1.28.tar.gz
-rw-r--r-- 1 doudou doudou   2465302 10月  9 04:00 redis-6.2.5.tar.gz

启动进程

[doudou@hadoop102 module]$ cd flink-1.13.0/
[doudou@hadoop102 flink-1.13.0]$  bin/start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host hadoop102.
Starting taskexecutor daemon on host hadoop102.
[doudou@hadoop102 flink-1.13.0]$ jps
12072 Jps
11769 StandaloneSessionClusterEntrypoint
12044 TaskManagerRunner

修改conf的masters和workers

[doudou@hadoop102 flink-1.13.0]$ cat conf/workers 
hadoop103
hadoop104
[doudou@hadoop102 flink-1.13.0]$ cat conf/masters 
hadoop102:8081

集群启动

分发

[doudou@hadoop102 flink-1.13.0]$ xsync flink-1.13.0/

启动

[doudou@hadoop102 flink-1.13.0]$ ./bin/start-cluster.sh 
Starting cluster.
Starting standalonesession daemon on host hadoop102.
Starting taskexecutor daemon on host hadoop103.
Starting taskexecutor daemon on host hadoop104.

向集群提交作业

可以引入插件 maven-assembly-plugin 进行打包。在 FlinkTutorial 项目的 pom.xml 文件中添加打包插件的配置

 <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

向命令行提交

传jar包

开放hadoop102

在hadoop103上开启

[doudou@hadoop103 flink-1.13.0]$ ./bin/flink run -m hadoop102:8081 -c com.atguigu.wc.StreamWordCount -p 2 ./FinkTutorial-1.0-SNAPSHOT.jar

关闭释放空间

[doudou@hadoop103 flink-1.13.0]$ ./bin/flink cancel c2e89d6899b1300ecde1cad336165ea9

YARN模式

配置环境变量

[doudou@hadoop102 flink-1.13.0]$ sudo vim /etc/profile.d/my_env.sh

HADOOP_HOME=/opt/module/hadoop-2.7.5
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export HADOOP_CLASSPATH=`hadoop classpath`

会话模式部署

执行脚本命令向 YARN 集群申请资源，开启一个 YARN 会话，启动 Flink 集群

bin/yarn-session.sh -nm test

提交作业

[doudou@hadoop102 flink-1.13.0]$ ./bin/flink run -c com.atguigu.wc.StreamWordCount ./FinkTutorial-1.0-SNAPSHOT.jar

单作业模式

执行命令提交作业

bin/flink	run	-d	-t	yarn-per-job	-c	com.atguigu.wc.StreamWordCount
FlinkTutorial-1.0-SNAPSHOT.jar