环境JDK1.8,开发语言JAVA,构建工具Maven3.6.1(阿里云镜像),Flink1.10.0
Kafka 自行安装,本次主要完成从kafka发送数据,flink接受数据进行处理,在输出处理结果
运行的代码
package com.xgx.flink;
import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011;
import java.util.Properties;
public class FlinkTest1 {
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
Properties properties = new Properties();
properties.setProperty("bootstrap.servers", "192.168.1.238:9092");
properties.setProperty("zookeeper.connect", "192.168.1.238:2181");
properties.setProperty("group.id", "smzGroup");
properties.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
properties.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
properties.setProperty("auto.offset.reset", "latest");
DataStreamSource<String> dataStream = env.addSource(new FlinkKafkaConsumer011<String>("test3", new SimpleStringSchema(), properties));
dataStream
.map(line -> Tuple2.of(line, 1))
.returns(Types.TUPLE(Types.STRING, Types.INT))
.keyBy(0).sum(1)
.print("dataStream: ");
env.execute("run job");
}
}
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.xgx.flink</groupId>
<artifactId>FlinkTest</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>1.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.12</artifactId>
<version>1.10.0</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka-0.11_2.12</artifactId>
<version>1.10.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<archive>
<manifest>
<mainClass>com.xgx.flink.FlinkTest1</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
主要代码片段就是这么多,在本地跑没有任何问题,点击package进行打包,选择下面那个进行使用
Flink计算环境部署,先要去官网下载
地址:https://flink.apache.org/zh/downloads.html
Flink 1.10.0以上的不能在window上部署,没有进行使用,我们点击进去下载flink-1.10.0-bin-scala_2.12.tgz
下载解压在bin目录下点击start-cluster.bat启动
在浏览器8081端口就可以打开WEBUI界面
我在使用Flink1.10.0程序在本地是可以使用的,在后台任务执行时却无法运行,发现时TaskManager无法启动,在conf/flink-conf.yaml 下配置就可以
taskmanager.cpu.cores: 2
taskmanager.memory.task.heap.size: 512m
taskmanager.memory.managed.size: 512m
taskmanager.memory.network.min: 64m
taskmanager.memory.network.max: 64m
然后重启后再次运行时还遇到了java.lang.ClassCastException LinkedMap
也在conf/flink-conf.yaml 进行配置
classloader.resolve-order: parent-first
然后重启再次运行就没问题了。
上面的问题在Flink1.9.3与1.9.2上都没有发现,自己直接启动服务就可以使用了。
附上效果图
在摸石头过河的路上越走越远,记录摸过的每一块石头。
Flink在每次重启后都会丢失之前提交的jar包,在conf/flink-conf.yaml 中配置自己的路径配置项即可
web.upload.dir: C:\Users\fenghua\Desktop\Flink\flink-1.10.0-bin-scala_2.12\flink-1.10.0-bin-scala_2.12\target
然后我们在每一次上传jar包后在重启都不会丢失之前提交的jar包