接着上面一节中介绍的,浏览器中输入地址http://swarm-manager:8088/cluster
,输出如下:
拿到application_1569070146084_0001,然后在控制台先kill掉:./yarn application -kill application_1569070146084_0001
Run a single Flink job on YARN
之前介绍了在一个Hadoop Yarn 环境中启动一个Flink集群,此外,还可以仅仅通过执行一个任务来启动一个flink
./bin/flink run -m yarn-cluster -yn 1 ./examples/batch/WordCount.jar
其中-m yarn-cluster
表示使用yarn集群,-yn 1
表示taskmanager的数量。
这个任务提交后,很快就可以运行完成:
自定义一个任务,并提交到Flink集群中
写一个最简单的代码
public class JavaHDFSBatchApp {
public static void main(String[] args) throws Exception {
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSource<String> dataSource = env.readTextFile("hdfs://swarm-manager:9000/LICENSE-2.0.txt");
dataSource.print();
}
}
在原有的maven基础上修改pom.xml文件,指定main class:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.0.0</version>
<executions>
<!-- Run shade goal on package phase -->
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.apache.flink:force-shading</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>org.slf4j:*</exclude>
<exclude>log4j:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<!-- Do not copy the signatures in the META-INF folder.
Otherwise, this might cause SecurityExceptions when using the JAR. -->
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.vincent.course08.JavaHDFSBatchApp</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
maven install
生成文件springboot-flink-train-1.0-shaded.jar
,导入到/home/iie4bu/app/flink-1.8.2/examples/batch
下,然后运行命令:./bin/flink run -m yarn-cluster -yn 1 ./examples/batch/springboot-flink-train-1.0-shaded.jar