大数据入门--Flink（二）流处理API

概述

• Environment
• Source
• Transform
• Sink

Environment

getExecutionEnvironment

创建一个执行环境，表示当前执行程序的上下文。 如果程序是独立调用的，则此方法返回本地执行环境；如果从命令行客户端调用程序以提交到集群，则此方法返回此集群的执行环境，也就是说，getExecutionEnvironment 会根据查询运行的方式决定返回什么样的运行环境，是最常用的一种创建执行环境的方式。

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

createLocalEnvironment

返回本地执行环境，需要在调用时指定默认的并行度。

LocalStreamEnvironment env = StreamExecutionEnvironment.createLocalEnvironment(1);

createRemoteEnvironment

返回集群执行环境，将 Jar 提交到远程服务器。需要在调用时指定 JobManager
的 IP 和端口号，并指定要在集群中运行的 Jar 包。

StreamExecutionEnvironment env = 
StreamExecutionEnvironment.createRemoteEnvironment("jobmanage-hostname", 6123, "YOURPATH//WordCount.jar");

Source

官方–datastream–conntectors

pom引入flink基础依赖

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java_2.11</artifactId>
    <version>1.13.2</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_2.11</artifactId>
    <version>1.13.2</version>
</dependency>

从集合读取

@Test
public void testFromCollection() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    DataStreamSource<String> ds = env.fromCollection(Arrays.asList("aa", "bb", "cc", "dd"));
    ds.print();
    env.execute();
}

从本地文件读取

public voidtestFromReadFile() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    URL resource = SourceTest.class.getResource("/SourceTest.txt");
    String path = resource.getPath();
    path = URLDecoder.decode(path, "UTF-8");
    DataStreamSource<String> ds = env.readTextFile(path).setParallelism(1);
    ds.print().setParallelism(1);
    env.execute();
}

从kafka读取

pom引入

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka_2.11</artifactId>
    <version>1.13.2</version>
</dependency>

@Test
public void testFromKafka() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    String brokers = "hadoop101:9092";
    KafkaSource<String> source = KafkaSource.<String>builder()
            .setBootstrapServers(brokers)
            .setTopics("flink-source")
            .setGroupId("my-group")
            .setStartingOffsets(OffsetsInitializer.earliest())
            .setValueOnlyDeserializer(new SimpleStringSchema())
            .build();

    DataStreamSource<String> ds = env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source");
    ds.print();

    env.execute();
}

自定义数据源读取

自定义 source。需要做的，只是传入
一个 SourceFunction 就可以。具体调用如下：

DataStream<String> dataStream = env.addSource( ...);

Transform

转换算子

Map

FlatMap

Filter

KeyBy

滚动聚合算子

sum()/max()/min()

maxBy()/minBy()

Reduce

Split 和 Select

Connect 和 CoMap

Union

富函数RichXXX

可以得到生命周期回调，比匿名函数的灵活性更高

Sink

官方–datastream–conntectors