flink入门

 

测试socketWindowWordCount example

win10端安装下载netcat(https://eternallybored.org/misc/netcat/netcat-win32-1.12.zip)，将可执行文件加入环境变量Path

执行命令 nc -l -p 9000   （监听9000端口，如果联通可以发送消息）【linux上直接 nc -l 9000即可】

# 启动集群
./bin/start-cluster.sh

nc -l 9000  # 启动netcat，向9000发送消息

# 运行flink example
./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000

# 参数含义
taskmanager.heap.size    # flink社区版的该参数是worker的总内存，包括堆内存和direct memory（Network Buffer）

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.kylinxue</groupId>
  <artifactId>FlinkExample</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <flink.version>1.7.0</flink.version>
    <scala.version>2.11.12</scala.version>
    <scala.binary.version>2.11</scala.binary.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-scala_2.11</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.47</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-connector-kafka-0.10_${scala.binary.version}</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-scala_2.11</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>redis.clients</groupId>
      <artifactId>jedis</artifactId>
      <version>2.10.2</version>
    </dependency>
  </dependencies>

</project>

 

Flink核心概念

flink容错机制：分组标记栏barrier    checkpoint

flink乱序事件流：通过watermark

 

Core API ：source --> transform --> sink

DataStream API

 

DataSet API

 

DataType和序列化

 

broadcast、accumulator、counter、distributed cache

 

状态管理

 

window和time