flink入门

最新推荐文章于 2024-06-03 15:28:52 发布

kylin_xue

最新推荐文章于 2024-06-03 15:28:52 发布

阅读量406

点赞数

分类专栏：大数据开发

本文链接：https://blog.csdn.net/kylin_xue/article/details/94125183

版权

大数据开发专栏收录该内容

1 篇文章 0 订阅

订阅专栏

测试socketWindowWordCount example

win10端安装下载netcat(https://eternallybored.org/misc/netcat/netcat-win32-1.12.zip)，将可执行文件加入环境变量Path

执行命令 nc -l -p 9000 （监听9000端口，如果联通可以发送消息）【linux上直接 nc -l 9000即可】

# 启动集群
./bin/start-cluster.sh

nc -l 9000  # 启动netcat，向9000发送消息

# 运行flink example
./bin/flink run examples/streaming/SocketWindowWordCount.jar --port 9000

# 参数含义
taskmanager.heap.size    # flink社区版的该参数是worker的总内存，包括堆内存和direct memory（Network Buffer）

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.kylinxue</groupId>
  <artifactId>FlinkExample</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <flink.version>1.7.0</flink.version>
    <scala.version>2.11.12</scala.version>
    <scala.binary.version>2.11</scala.binary.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-scala_2.11</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.47</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-connector-kafka-0.10_${scala.binary.version}</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-scala_2.11</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
      <version>${flink.version}</version>
    </dependency>
    <dependency>
      <groupId>redis.clients</groupId>
      <artifactId>jedis</artifactId>
      <version>2.10.2</version>
    </dependency>
  </dependencies>

</project>

Flink核心概念

flink容错机制：分组标记栏barrier checkpoint

flink乱序事件流：通过watermark