实验环境
JDK 1.8
IDE Intellij idea
Flink 1.8.1
实验内容
创建一个Flink简单Demo,可以从流数据中统计单词个数。
实验步骤
首先创建一个maven项目,其中pom.xml文件内容如下:
<properties>
<flink.version>1.8.1</flink.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-wikiedits_2.11</artifactId>
<version>${flink.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.1.4.RELEASE</version>
<configuration>
<mainClass>wikiedits.StreamingJob</mainClass>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
创建一个包com.vincent,并且创建一个类StreamingJob.java
public class WikipediaAnalysis {
public static void main(String[] args) throws Exception {
}
}
Flink 程序的第一步是创建一个StreamExecutionEnvironment。StreamExecutionEnvironment可以设置参数并且导入一些外部系统的数据源。
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
接下来创建一个外部数据源,外部数据源使用nc -l 9000 表示服务器端开启监听9000端口,并可以发送数据。
DataStream<String> text = env.socketTextStream("192.168.152.45", 9000);
这样就添加了一个流文本数据源,有了DataStream就可以获取数据了,然后对数据进行分析:
DataStream<Tuple2<String, Integer>> dataStream = text.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
@Override
public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
String[] tokens = s.toLowerCase().split("\\W+");
for (String token : tokens) {
if (token.length() > 0) {
collector.collect(new Tuple2<String, Integer>(token, 1));
}
}
}
}).keyBy(0).timeWindow(Time.seconds(5)).sum(1);
flatMap表示将嵌套集合转换并平铺成非嵌套集合,字符串是s,返回值是Collector<Tuple2<String, Integer>>。并且根据keyBy(0)即第0个字段进行统计加一操作。.timeWindow()指定窗口大小是5秒。
所以整体代码如下:
public class StreamingJob {
public static void main(String[] args) throws Exception {
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> text = env.socketTextStream("192.168.152.45", 9000);
DataStream<Tuple2<String, Integer>> dataStream = text.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
@Override
public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
String[] tokens = s.toLowerCase().split("\\W+");
for (String token : tokens) {
if (token.length() > 0) {
collector.collect(new Tuple2<String, Integer>(token, 1));
}
}
}
}).keyBy(0).timeWindow(Time.seconds(5)).sum(1);
dataStream.print();
// execute program
env.execute("Java WordCount from SocketTextStream Example");
}
}
运行
运行main方法,然后在服务器端执行nc -l 9000 并且输入文本:
iie4bu@swarm-manager:~$ nc -l 9000
a b d d e f
然后在intellij控制台将输出:
1> (b,1)
3> (a,1)
1> (f,1)
3> (d,2)
1> (e,1)
可以统计出每个单词的次数