一、架构说明(standalone模式)
standalone是Flink自带的一个分布式集群,它不依赖其他的资源调度框架、不依赖yarn等。Spark也有一个自带的分布式资源调度模型,也叫作standalone
standalone模式进程名介绍:
- JobManager------->进程名字:
StandaloneSessionClusterEntrypoint
我们又叫它:Master
- TaskManager----->进程名字:
TaskManagerRunner
我们又叫它:Worker
1.1 机器分布
机器名(hosts) | IP | 角色 |
---|---|---|
bigdata111 | 192.168.1.111 | JobManager、TaskManager |
bigdata112 | 192.168.1.112 | TaskManager |
bigdata113 | 192.168.1.113 | TaskManager |
1.3 搭建步骤
-
上传 Flink 安装包至 Linux 服务器 master 节点。使用 tar 命令解压安装包
tar -zxvf flink-1.7.2-bin-scala_2.12.tgz -C /home/xyp9x/
-
修改 conf 目录下的
flink-conf.yaml
配置文件vi flink-conf.yaml # 指定jobmanager的地址(此处可使用ip/别名) jobmanager.rpc.address: bigdata111 # 指定taskmanager的可用槽位的数量(槽位越多,运行的subTask就越多) taskmanager.numberOfTaskSlots: 2
-
修改 conf 目录下的
masters
配置文件,指定jobmanager
的所在节点vi masters bigdata111
-
修改 conf 目录下的
slaves
配置文件,指定taskmanager
的所在节点vi slaves bigdata111 bigdata112 bigdata113
-
环境变量
vi /etc/profile export FLINK_HOME=/home/xyp9x/flink-1.7.2 export PATH=$PATH:$FLINK_HOME/bin source /etc/profile
-
将配置好的 Flink ,拷贝到其他节点(112,113节点)
scp -r flink-1.7.2/ xyp9x@bigdata112:`pwd` scp -r flink-1.7.2/ xyp9x@bigdata113:`pwd`
-
配置完成,目前环境配置为:
1个JobManager,3个 TaskManager
1.4 启动Flink集群
start-cluster.sh
1.5 访问JobManager 的 Web 管理界面
192.168.1.111:8081
二、搭建 maven 工程
<dependencies>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-scala_2.12</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-scala_2.12</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka_2.12</artifactId>
<version>1.7.2</version>
</dependency>
<dependency>
<groupId>org.apache.bahir</groupId>
<artifactId>flink-connector-redis_2.11</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.44</version>
</dependency>
</dependencies>
<build>
<plugins>
<!-- 该插件用于将 Scala 代码编译成 class 文件 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.4.6</version>
<executions>
<execution>
<!-- 声明绑定到 maven 的 compile 阶段 -->
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
无法加载类时build可以换另外一个:
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.19</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>