Apache Flink 零基础入门（三）编写最简单的helloWorld

实验环境

JDK 1.8

IDE Intellij idea

Flink 1.8.1

实验内容

创建一个Flink简单Demo，可以从流数据中统计单词个数。

实验步骤

首先创建一个maven项目，其中pom.xml文件内容如下：

    <properties>
        <flink.version>1.8.1</flink.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-scala_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-wikiedits_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.1.4.RELEASE</version>
                <configuration>
                    <mainClass>wikiedits.StreamingJob</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>

创建一个包com.vincent，并且创建一个类StreamingJob.java

public class WikipediaAnalysis {
	public static void main(String[] args) throws Exception {

	}
}

Flink 程序的第一步是创建一个StreamExecutionEnvironment。StreamExecutionEnvironment可以设置参数并且导入一些外部系统的数据源。

final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

接下来创建一个外部数据源，外部数据源使用nc -l 9000 表示服务器端开启监听9000端口，并可以发送数据。

DataStream<String> text = env.socketTextStream("192.168.152.45", 9000);

这样就添加了一个流文本数据源，有了DataStream就可以获取数据了，然后对数据进行分析：

        DataStream<Tuple2<String, Integer>> dataStream = text.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                String[] tokens = s.toLowerCase().split("\\W+");

                for (String token : tokens) {
                    if (token.length() > 0) {
                        collector.collect(new Tuple2<String, Integer>(token, 1));
                    }
                }
            }
        }).keyBy(0).timeWindow(Time.seconds(5)).sum(1);

flatMap表示将嵌套集合转换并平铺成非嵌套集合，字符串是s，返回值是Collector<Tuple2<String, Integer>>。并且根据keyBy(0)即第0个字段进行统计加一操作。.timeWindow()指定窗口大小是5秒。

所以整体代码如下：

public class StreamingJob {
    public static void main(String[] args) throws Exception {
        final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStream<String> text = env.socketTextStream("192.168.152.45", 9000);
        DataStream<Tuple2<String, Integer>> dataStream = text.flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
            @Override
            public void flatMap(String s, Collector<Tuple2<String, Integer>> collector) throws Exception {
                String[] tokens = s.toLowerCase().split("\\W+");

                for (String token : tokens) {
                    if (token.length() > 0) {
                        collector.collect(new Tuple2<String, Integer>(token, 1));
                    }
                }
            }
        }).keyBy(0).timeWindow(Time.seconds(5)).sum(1);

        dataStream.print();
        // execute program
        env.execute("Java WordCount from SocketTextStream Example");
    }
}

运行

运行main方法，然后在服务器端执行nc -l 9000 并且输入文本：

iie4bu@swarm-manager:~$ nc -l 9000
a b d d e f

然后在intellij控制台将输出：

1> (b,1)
3> (a,1)
1> (f,1)
3> (d,2)
1> (e,1)

可以统计出每个单词的次数

转载于:https://my.oschina.net/duanvincent/blog/3098336